Deduplication (data optimization) is a hot topic these days because its promise to reduce storage consumption and its associated costs is compelling to today’s high data growth businesses. Impact on IT budgets, CAPEX and OPEX as a result of applying dedupe in primary and/or secondary tiers of storage can be substantial!

Lost in all the hype is the fact that there are as many flavors of dedupe as Ben & Jerry’s has flavors of ice cream! They are all different and provide varied results. There are a few vectors to pivot around in this discussion such as file vs. sub-file, memory efficiency and scalability.   Each are part of the puzzle vendors bring together to help reduce the amount of data and associated storage space consumed. Unfortunately, these puzzle pieces determine the effectiveness of the dedupe offering yielding a high performance solution vs. a minimal capability solution.  Just like a sports car compared to a VW Beetle. Both will provide a service, it’s a matter of the results!

In each characteristic, the impact and yield really make a difference;

* File vs. Sub-file - as data is analyzed the smaller the chunk size the higher the deduplication yield. So deduping at the file level yields the least dedupe benefit while deduping at the smaller (sub-file) data chunk level (4K is optimal) yields the largest benefit.

* Memory efficiency is critical since the more hash keys you can keep in memory the better. The closer you can function to 100% ‘in memory,’ the better the overall performance will be!

* Not too far behind memory efficiency is the ability to sample the most amount of data. Again, the more efficient the better and the higher the scalability will be.

Here’s a case in point. Permabit’s Albireo High Performance Data Optimization solution recently changed the deduplication market as it applies to primary storage data optimization (dedupe) and has received substantial press and analyst recognition.  Contrasting Albireo is the Open ZFS Deduplication technology which has been used as a base for some storage optimization solutions. Let’s compare the two:

* ZFS uses a 128K chunk size vs. Albireo using as small as a 4K chunk (it’s configurable).  Albireo will find substantially more duplicates and reduce storage consumption dramatically compared to ZFS

* ZFS uses a table entry approach for data identification at 200 bytes each vs. Albireo using an index with data representation of only 4 bytes each.  Albireo will typically reside in memory 99.5% of the time. ZFS on the other hand, will page out to disk (resulting in major performance slow down) for table look-up more than 50% of the time as a result of the 200 byte data id (50 times the size of Albireo).

* ZFS can scale out to 20TB using 32 GB of RAM vs. Albireo which can scale to 640TB using only 16GB of RAM (64 times greater scalability with Albireo)!

Albireo will be more effective at finding duplicates, more efficient at processing the data and more scalable.   The contrast is significant because businesses do run on the data that is being deduped and the impact on the overall storage costs will be substantial. The higher performance, the more efficient and the more scalable one flavor of dedupe is over another, the better! Like Cherry Garcia is over all those other flavors, Albireo is to the rest of the dedupe offerings!

A few comments have been made regarding latency, most recently in a Search Storage piece from Dave Raffo, when comparisons were being made between compression and deduplication applied to primary storage. The concern expressed is that deduplication may be inflicting latency in the write or read process for primary storage. That may have been an astute observation a year ago but today its simply just not a valid observation!

At issue is the performance of primary storage which needs to be as efficient as possible since businesses are run on primary storage that houses transactional data for core applications such as finance, manufacturing, CRM, sales operations, and supply chain to mention just a few. For these applications any significant delay in data access is simply not tolerable because businesses run on them! With today’s high performance data optimization (deduplication) tools such as Permabit’s Albireo there is no latency on data access because Albireo is NOT IN THE READ PATH!  And for that matter Albireo is not in the write path for all but in inline implementations.

In a recent blog post NOT in the READ Path I go into some of the details ‘As we designed and developed Albireo, we specifically addressed the most critical of all performance criteria - data read. Why is it significant? Because data is usually written once, but can be read hundreds, thousands or millions of times clearly affecting the performance of the storage.  As a result, Albireo is NOT in the read path — so there is no performance impact on read and no latency!

In the write process Albireo’s flexibility enables inline, parallel and post-process approaches (or any combination of the three) so the storage vendor can choose which approach they prefer. Let’s look at each of these:

* Of course in an inline approach, there is a very small write latency as Albireo does a hash key create and look-up. When we say very small, we’re talking a few microseconds. Can a user detect a few microsecond delay on write?  I think not!

* In a parallel approach. there is NO performance impact on write at all because the vendor continues to write the data as they normally do.  Albireo provides duplicate advice to the storage and the vendor can apply it when cycles permit.

* In a post-process deduplication approach, there is NO performance write impact because a write is done as the vendor normally would and a post-process deduplication can be initiated as cycles are available.’

Albireo is a new approach to data deduplication that is embedded into the storage stack using patented indexing, memory utilization and hash key technologies enabling scale out and performance that until Albireo has not been seen in other deduplication implementations.  Previous iterations of dedupe, which have primarily been used for backup, had their idiosyncrasies such as performance, latency, scale-out and memory usage to name but a few. These do not apply in Albireo’s implementation because we have solved these issues as we developed Albireo.

In summary, using Albireo High Performance Data Optimization (deduplication) in primary storage does not introduce latency!

Albireo addresses the data safety issue head on. It never changes the data! Unlike a point solution or appliance, Albireo also doesn’t change the data or impact the read/write process (see NOT in the Read Path) because it is an advisory service to the storage stack. Simply said, Albireo analyzes the data, determines if it is duplicate and advises the storage that it is or is not a duplicate. If it is not a duplicate, the storage system does nothing to the data. If it is a duplicate, the storage system can create a pointer to the original data and remove the duplicate releasing storage space and reducing associated costs. The effect is that there will be less storage consumed, less power/cooling needed, less data center footprint and less storage to be bought to meet anticipated data growth.  These impacts drop to your CAPEX and OPEX providing budget relief!

How does this impact data safety? Simple, the data is written as it always would be by the storage system in its native. It is never changed or modified nor does it need an interpreter to reconstitute it.  There is no third party dependency! So if the data optimization functionality is disabled, removed, or fails data can still be read. Your data is as safe as it always is and is protected by your storage system!

Our partners will be deploying it in storage systems later this year and next year. If your storage vendor is not discussion this technology advance with you ask them about it!

Albireo is a technological step forward. It will impact your business in a very positive way and it will be easier, more efficient, eliminate risk and be less costly as it optimizes the data storage your business runs upon. Isn’t that what progress is all about!

‘Don’t impact performance’ I addressed in my blog last week: ‘Not in The Read Path’. Simply said, we don’t impact performance and provide a simple and elegant method of implementing deduplication. As a result of our approach, we developed a dedupe advisory approach that also enables the OEM to utilize all of their existing features as well!

When it comes to flexibility, we really took a step forward. We knew the concerns about deduplication technology revolved around 3 key areas:

* Hash key development/indexing and performance impact

* Resource utilization

* Deployment flexibility.

Hash keys are the lifeblood of deduplication in that they are the unique identifier of a data chunk. Once the key is developed for a chunk of data the next question is ‘have I seen this data before?’ This is where it gets difficult because doing this look up is what causes most dedupe systems to come to a grinding halt. Big indexes and large keys (yes we use SHA 256 so the key is large) create performance slow downs and in some lesser systems, scalability limits.  We skinned that cat with patented indexing that can return an index lookup in a few micro seconds! Hardly impact the performance at all! And scale out is also managed by patented memory based indexing that is 99.5% memory resident! No slow down there either!  Hardly ever see a disk fetch!   Once we do this we return advice to the OEM storage stack that the data chunk is a duplicate. The OEM then creates a pointer for the duplicate.

Resource utilization or how much of the processing power is consumed is a significant issue for storage vendors.  For example, newer systems have fast quad core processors (sometimes 2!)  while older or less high performance systems may have dual core processors.  In any case a flexible dedupe engine should enable the OEM to define the number of processors they used for Albireo depending on their overall system performance criteria. We designed that flexibility into Albireo to ensure our deduplication did not consume resources that would impact our partners performance.

Flexibility as you can see from the resource utilization comments is already there but we took an additional step by enabling our OEM partners to deploy Albireo in inline, post-process or parallel (hybrid) methods.  For example, if the partner performance tests enable an inline deduplication within the OEMs performance characteristics then they can implement Albireo inline!   If they want to balance their performance and deduplication efficiency then parallel mode may be the best for that partner. And finally, if high performance storage were the dominant criteria, then post-process mode for Albireo would work for that partner! We also give the option for the OEM to leverage all 3 so depending on use cases, they can implement the appropriate method (all within one system). Their choice!

All this is done with an SDK that can be integrated with 6-8 API calls. Some of our partners have integrated in as little as a few days. Albireo is an OEM partners tool that they can deploy based on their performance, resource and efficiency design criteria.  We made it that way!

In my next posting I’ll discuss data protection.

The most significant concern raised as we discussed Albireo with our partners has been the potential impact to performance. In many ways it’s their stock and trade and they protect it as any business should ‘with their lives!’ The thought of embedding deduplication technology, even though there is a clear customer benefit in cost, space and power/cooling reduction, is orthogonal to their performance mantra particularly in primary storage.

As we designed and developed Albireo, we specifically addressed the most critical of all performance criteria - data read. Why is it significant? Because data is usually written once, but can be read hundreds, thousands or millions of times clearly affecting the performance of the storage.  As a result, Albireo is NOT in the read path — so there is no performance impact on read!

In the write process, we also were careful to provide our partners choices that enable them to determine whether to impact their performance or not.  Albireo’s flexibility enables inline, parallel and post-process approaches (or any combination of the three) so the storage vendor can choose which approach they prefer. Let’s look at each of these:

* Of course in an inline approach. there is a very small write latency as Albireo does a hash key create and lookup. When we say very small, we’re talking mere microseconds across the entire data stream. Can a user detect a microsecond delay?  I think not!

* In a parallel approach. there is NO performance impact on write at all because the vendor continues to write the data as they normally do.  Albireo provides duplicate advice to the storage and the vendor can apply it when cycles permit.

* In a post-process deduplication approach. there is NO performance write impact because a write is done as the vendor normally would and a post-process deduplication can be initiated as cycles are available.

In summary, Albireo does NOT impact performance on read! There is also NO impact to write performance in parallel or post-process modes.  So, the vendor concerns about performance are addressed!

In my next post I’ll address implementation simplicity.

Usually summer is a time for reflection and sun tanning at the beach. From a business perspective things tend to slow down a bit while each of us takes a second breath and spend some quality time with our families away from the business side of our lives. There is usually not much industry news few ‘big deals’ occur that were not already in play. Technology shift announcements are typically saved for September when everyone is back, so there is as much traction as possible. Not the case for the summer of 2010!

It may have all started in early June when Permabit announced Albireo. Albireo is a simple and elegant data optimization technology that makes it possible to embed data deduplication into primary storage arrays with a few API calls to the existing storage firmware/operating environment.  The press, analysts, OEMs and bloggers,  were very supportive and effusive: ‘the tectonic plates of storage have shifted’ or ‘Killer dedupe’ or ‘This stuff is so far ahead in its capabilities and performance I can’t see why you (OEMS) would want to do it yourself’ to mention a few.  And yes, there were competitors snipes in the blogosphere - few were given any credence because of the support from press, analysts and OEMs. This started a domino effect that continues today.

A few weeks later, HP announced StoreOnce at their conference and talked about end-to-end deduplication (data optimization) setting a great vision! They went on to announce and describe their implementation of HP Labs deduplication technology (StoreOnce) via their D2D backup appliance. The rest of the details will evolve as they deliver and fulfill the end-to-end vision over the next 18-24 months.

Not to be upstaged by either a startup from Cambridge or the second largest storage vendor on the planet, EMC said ‘We got that’ as they announced their ‘Viper Team’ was working on data optimization (deduplication & compression) technologies that would concatenate the IP within EMC and offer an ability to apply end-to-end data optimization on their entire storage offering - another vision of the end state.  Details to be announced later!

Rumors were flying by then involving IBM and Dell acquisitions. . Dell would eventually buy Ocarina for their IP in compression and deduplication via Kleiner-Perkins with the implied intent to deploy data optimization across their storage offering. They also want to become a storage supplier beyond their own offerings by hoping to engage with the OEM partners that Ocarina had partnered with.

IBM would announce their acquisition of Storwize whose compression IP could eventually be embedded into IBM storage. This would be even more powerful if combined with deduplication (isn’t that data optimization too!) and reduce storage consumption for their customers.

All that in June and July!  What will August bring?

For end customers ‘this changes everything!’  Why you might ask?  Data stored in primary data stores will be compressed and/or deduplicated saving 50-75% or more of the storage space required. As a result, effective costs of storage will be substantially reduced, data center floor space requirements will drop proportionally, cooling and power will do the same and resources to manage ever expanding data stores will become more efficient. In parallel, storage drives will continue to become denser and 2, 4 and 6TB drives will be seen in storage array deployments during 2010 and 2011. Storage vendors will also begin deployment of more efficient unified storage technologies that incorporate embedded deduplication and/or compression, while legacy storage may be enhanced to incorporate embedded deduplication and/or compression. As end-to-end data optimization (extending the impact of dedupe/compression to secondary, archive and backup storage) is deployed, the cost reductions mention above regarding primary storage will multiply and storage expenses will drop significantly as a proportion of IT budgets! Bottom line - your CAPEX and OPEX will be significantly reduced as a result. That’s why the events of the summer of 2010 changed everything!

If your storage supplier is not telling you they have or will have data optimization in their primary storage offerings, if not end-to-end, I’d ask why not and reconsider your storage vendor choices!

Incumbent vendors have a huge opportunity because of the thousands of Mikes out there. And a few have the formula right! They can and do optimize their time to market by not sticking to the typical “not invented here” syndrome.  The best storage companies have shown an ability to advance products with a combination of internal, OEM and strategic acquisitions as sources for the “kit” they offer. For example, more enlightened vendors optimize by “sticking to their knitting” doing what they do best, build hardware and operating software that optimizes their product offerings. Once they establish base level architecture and systems, they source features from the best-of-breed suppliers (yes Mike those smaller companies!) for advanced efficiency, automated tiering, performance, for example, to embellish and differentiate their offerings. The result is that the enlightened incumbent vendor can deliver products to market faster and more efficiently, that is differentiated from their competitors and more closely addresses their customers “pain.” The result is that the Mikes of the world can have the security of the incumbent vendor with rapid time-to-market solutions to address their needs. You just have to make sure you pick the right ones!

Because there are aspiring companies who have advanced efficiency, automated tiering, performance, data optimization and deduplication, the pressure has never been higher to obtain the best-of-breed and embed it into the incumbent vendor offerings.  Today there is increasing momentum to build out the next generation of storage solutions that optimize feature sets, differentiate and deliver quickly. A few have it right!   Mike has a point!

For vendors to implement Dedupe 2.0, they must address the different use cases. For example, archiving is more about data volume with the amount of data increasing rapidly and also being stored for increasingly longer periods of time. In tiered storage, the concerns are about data volume and to some degree speed of ingest and data retrieval. Primary storage is the most difficult because it’s really all about implementing dedupe with zero impact on storage performance on the ingest and retrieval activities.

Dedupe 2.0 advances are being enabled by two technologies that are converging today. First is processing capabilities (multiple quad core) and second, and probably the most important, is the ability to very rapidly query an index to determine if there is a duplicate.

The ability to very rapidly query an index of digital fingerprints consumes processor cycles that until recently inhibited near real-time deployment. The solution required the development of indexing techniques that use memory to house the index. By doing so the query can be accelerated and dedupe would not impact ingest or rehydrate efforts and not impact system and storage throughput. Permabit developed and patented indexing techniques, over the last decade, which can return index queries in microseconds. We have been shipping this technology for the last several years in our Enterprise Archive product.

So the answer is that Dedupe 2.0, is the deployment of deduplication technology higher up in the storage stack. In primary, tier 2 and archive. Addressing duplicates before the data ever becomes a backup problem! Basically, dedupe everywhere.

It was just a matter of time! The Dedupe 1.0 era comes to an end! For readers that may not know the phrase Dedupe 1.0 is what we refer to as the initial phase of dedupe which employed the use case of dedupe in backup. Simple enough backups by their nature contain an excessively high duplicate rate. Since the daily backup is 90% duplicate from the previous day it stands to reason that the initial use case for dedupe would be backup.

Unfortunately, the environment (backup) dictated the design and application of deduplication. For example, backups are not usually petabytes or more in size so the design was for smaller data sets. In addition, ingestion of the backup data needed to be as short as possible to minimize the impact on running applications. Some vendors just did a quick copy and later deduped the backup data. Dedupe uses hash key algorithms to create unique IDs for data so that dupes can be found. Applying a smaller algorithm such as MD5 or SHA1 delivered more processing efficiency. This would enable them to use less costly and slower processors, keeping the price points down or maybe enabling a bit more margin in the offering. Unfortunately that also exposed some of the data to possible corruption because the smaller hash keys were subject to data collisions meaning there would be two keys that were the same for different data! OOPS!

Those days are behind us. The advent of quad core processors that can be deployed in pairs provided enough compute capability to enable inline deduplication to occur eliminating the post process of data and to even enable SHA 256 hashing to occur resulting in data integrity. The result of these advances is that dedupe is now ready to move beyond backup into tier 2 storage and finally primary storage. We are calling this Dedupe 2.0!

Dedupe 2.0 applies to archiving data in disk based archives, tier 2 data on less costly NAS architectures and even to primary data. Duplicates are in each of these data stores and the more efficient these data stores are, the less storage they consume, the less power/cooling they consume and the less floor space they require and finally in these IT budget challenged times the lower the cost to store that data!

I have been tracking market trends and buying patterns for the last year on this subject and am very glad to see the 2010 IT user survey in Search Storage on February 15. The survey across over 360 IT professionals in all market segments indicated the 77% are either deploying or evaluating deduplication in primary or tier 2-n storage in 2010. So 2010 will be a watershed year for deduplication almost 80% of IT professionals will take dedupe to the next level to Dedupe 2.0. Why because it will save them money, costs, resources and most importantly time and effort. The next step in deduplication is arriving this year!

Every New Year brings with it hope for a more prosperous year than the one before and a clearing of the previous year’s frustrations. We all enter New Years Eve with a sense of optimism and it’s a good time to look at the horizon and predict what we think the next be 12 months will look like.

Having waited a few weeks; I have tempered my unbridled optimism while looking ahead to the end of the year 2010. From a technology perspective, I continue to be optimistic, which in sharp contrast to what my views were for 2009. Last year was an incredibly difficult and frustrating year for everyone. The economy tanked and IT spending was off so markedly that some businesses, simply stopped spending on IT altogether. In most cases the IT spend was on an as needed basis and in all cases “if it doesn’t have a quick ROI” the money was just not available. At this point we seem to be past the worst of it and there are many signs that validate that opinion.

The business dynamics of the last year enabled a “New Normal” to evolve that required optimization of existing investments, tight budget controls and a “flight to efficiency” that we haven’t seen in IT for many years! These behaviors seem to have taken hold and will be the model for 2010 and beyond. Luckily, the technology industry advances are helping with lower cost storage as density increases and the cost per GB continues down a curve. Many of these advances have been from companies that did not stop investing in R&D and did not fall to the economic cycle of 2008/9. The investments were made by smaller more agile technology companies that are driving the trend back to a more prosperous business climate.

The years 2008/2009 saw rapid expansion and adoption of some key technologies such as deduplication, virtualization and cloud. Each has been fodder for the industry analyst communities and the technology press that have called each a new frontier and the savior of the IT budget! In fact, each is helpful and each has been impactful. However, they are at the early stages of their lifecycle and broad adoption is still in the future. Albeit near future in some cases!

Deduplication has been deployed in the backup space for the last few years and has risen to become a “must have technology.” It can save space, costs of storage and improve efficiency in the backup cycle! OK good first use case! But what about the rest of the information businesses manage? Primary storage, where the most active and important business information is stored must have duplicates? Or, how about tier 2, beyond storage, and archive storage? These all have duplicates! In 2010, use of deduplication will extend to all of these. The most difficult will certainly be primary storage where deduplication must be “non impactful” in the overall ingestion and rehydration of information. Technical advances from those smaller vendors I mentioned earlier will enable this to happen and broad adoption including primary, tier2 –n and archive will begin to occur in 2010!

Virtualization has also taken hold of the IT imagination and the seamless and transparent access that it employs has improved the efficiency of information management, user deployment and information complexity. The key enabler has been the ability to rapidly deploy virtual machines. One caveat in this panacea is the virtual machine images themselves. The backup/compliance/gold master copies of the images. They consume huge amounts of storage and for the most part are 90% the same. What a great opportunity for deduplication! As deduplication use cases expand in 2010 the bloat seen from VM images will be tamed!

Cloud computing and more specifically cloud storage seemed to be rapidly rising as an alternative model. The ability to shift CAPEX to OPEX during the economic cycle of 2009 was appealing to the finance managers who held the purse strings of many IT managers. However, historical awareness of the initial attempts of “storage as a service” several years ago quickly brought down to earth the reality of data security and public storage! In recent discussions with industry analysts and CIO surveys, the fear of security issues has quickly slowed the fervor on cloud deployments in many enterprises. 2010 will see the small steps forward in cloud computing that will deploy “private clouds” that will be within the corporate firewall but provide shared deployments enabled by multi-tenancy support. This will enable a better utilization of resources and optimize information management and CAPEX. In some cases, smaller businesses may utilize public cloud deployments as a necessity continuing to be driven by the economics they provide.

One necessary technology that I believe will quickly evolve is new data protection solutions for larger capacity drives. RAID is “running out of gas”! As we continue to see larger and larger data stores many in the 100’s of TB and even reaching petabytes. These data stores are utilizing the latest and largest drives available, 1 TB and 2 TB. As the scale grows, it is increasingly risky to depend on RAID technology that was initially developed in the 1980s! Back then a petabyte may have been mathematically possible but was not included in the design of RAID. Today petabyte data stores are reality and increasingly more probable as we continue to store and retain more information daily. The key issue is at the petabyte scale, using 1 and 2 TB drive technology; RAID cannot recover effectively and efficiently and without data loss or corruption if there is a failure. Newer more scaleable technologies using erasure coding techniques can scale and provide the recovery and data protection necessary in the petabyte scale data stores we will see more frequently in 2010 and beyond.

2010 will be an interesting year and I am optimistic that we will see the advances mentioned above take hold and deliver increased value and cost/efficiency to IT and a more robust economy. Let’s not forget what we have endured and let’s apply what we have learned to optimize our business and our information!

]]> http://blog.permabit.com/index.php/2010/02/2010-predictions/feed/