Albireo’s unique design provides efficient deduplication without any performance penalty because Albireo runs entirely out of the read path.  Albireo’s role is to provide deduplication advice to existing storage subsystems. Simply and elegantly, Albireo takes advantage of the built-in capabilities of modern file system and block virtualization layers provided by all mainstream storage vendors.  With this design, Albireo creates no more performance impact than when common features such as clones and snapshots are in use on the storage system.

So what about ingestion speeds?   This is where Albireo shines.   The grid-based capabilities of the Albireo index allow it to be scaled across multiple independent CPU resources to provide deduplication advice on ingest at breakneck speeds.  While Albireo is rated in excess of 140 MB/sec per processor core on a 3.0 GHz Intel Xeon even when operating with a small 4 KB chunk size, performance is even greater with larger chunk sizes. With 64KB chunks we’ve seen Albireo Grid-based dedupe scale to support ingest speeds over 5.5GB/s.   These speeds are more than adequate to meet the needs of even the most intensive VMware environments. So there is no performance impact or latency as has been identified in early stage deduplication backup offerings.

What’s more, Albireo allows storage vendors to give their customers choice.  With Albireo, an OEM can chose to implement  deduplication inline, in parallel, and/or as a post-process event - providing the greatest flexibility for performance intensive environments while simplifying manageability.  This means that in the latest powerful storage equipment deduplication can be performed inline with no performance impact, while in more resource constrained storage hardware, customers simply schedule deduplication to run at off hours.  Either way, there is no performance impact to the user environment.

Finally, I wanted to talk a little about a way in which deduplication can actually improve read performance.   While Albireo itself doesn’t sit in the read path, it does provide advice which allows blocks in the read path to be shared.   Like VMware Transparent Page Sharing (TPS), Albireo looks at new data and provides advice on how to eliminate duplicate blocks using existing capabilities of modern storage systems. These existing capabilities maintain deduplicated blocks in the read cache, reducing the amount of memory needed for caching the operating systems and applications stored in VMware environments.  The result is improved overall performance when reading these shared blocks from disk.   This becomes especially important in virtual desktop deployments, where many users logging in at the same time can result in boot storms.  In these environments, deduplication can reduce the impact of boot storms by up to 50%.

Deduplication (data optimization) is a hot topic these days because its promise to reduce storage consumption and its associated costs is compelling to today’s high data growth businesses. Impact on IT budgets, CAPEX and OPEX as a result of applying dedupe in primary and/or secondary tiers of storage can be substantial!

Lost in all the hype is the fact that there are as many flavors of dedupe as Ben & Jerry’s has flavors of ice cream! They are all different and provide varied results. There are a few vectors to pivot around in this discussion such as file vs. sub-file, memory efficiency and scalability.   Each are part of the puzzle vendors bring together to help reduce the amount of data and associated storage space consumed. Unfortunately, these puzzle pieces determine the effectiveness of the dedupe offering yielding a high performance solution vs. a minimal capability solution.  Just like a sports car compared to a VW Beetle. Both will provide a service, it’s a matter of the results!

In each characteristic, the impact and yield really make a difference;

* File vs. Sub-file - as data is analyzed the smaller the chunk size the higher the deduplication yield. So deduping at the file level yields the least dedupe benefit while deduping at the smaller (sub-file) data chunk level (4K is optimal) yields the largest benefit.

* Memory efficiency is critical since the more hash keys you can keep in memory the better. The closer you can function to 100% ‘in memory,’ the better the overall performance will be!

* Not too far behind memory efficiency is the ability to sample the most amount of data. Again, the more efficient the better and the higher the scalability will be.

Here’s a case in point. Permabit’s Albireo High Performance Data Optimization solution recently changed the deduplication market as it applies to primary storage data optimization (dedupe) and has received substantial press and analyst recognition.  Contrasting Albireo is the Open ZFS Deduplication technology which has been used as a base for some storage optimization solutions. Let’s compare the two:

* ZFS uses a 128K chunk size vs. Albireo using as small as a 4K chunk (it’s configurable).  Albireo will find substantially more duplicates and reduce storage consumption dramatically compared to ZFS

* ZFS uses a table entry approach for data identification at 200 bytes each vs. Albireo using an index with data representation of only 4 bytes each.  Albireo will typically reside in memory 99.5% of the time. ZFS on the other hand, will page out to disk (resulting in major performance slow down) for table look-up more than 50% of the time as a result of the 200 byte data id (50 times the size of Albireo).

* ZFS can scale out to 20TB using 32 GB of RAM vs. Albireo which can scale to 640TB using only 16GB of RAM (64 times greater scalability with Albireo)!

Albireo will be more effective at finding duplicates, more efficient at processing the data and more scalable.   The contrast is significant because businesses do run on the data that is being deduped and the impact on the overall storage costs will be substantial. The higher performance, the more efficient and the more scalable one flavor of dedupe is over another, the better! Like Cherry Garcia is over all those other flavors, Albireo is to the rest of the dedupe offerings!

We have two announcements at the show. The first is around Permabit extending our VMware Alliance to the Select level.  That means we are partnering more closely with VMware and their ecosystem. We are the only OEM data optimization partner of VMware.  Our second announcement is of our new OEM relationship with BlueArc. Interestingly, the VMworld theme of every storage vendor, save two, is about ‘interoperability’. While good, interoperability or integrating with the VM management console isn’t that exciting to me. The real value is in optimizing the storage of VM. That brings me back to my main point - only two storage companies are talking about optimizing VMware environments - NetApp and our new OEM partner BlueArc and that sets them apart from the rest of the storage crowd.

A couple of weeks ago the subject of how important data optimization is to NetApp’s growth came up during a discussion with a prominent analyst. He attributed NetApp’s growth to two factors: adopting primary data optimization and hitching their wagon to VMware. In effect he said, by implementing a dedupe solution and becoming the preferred storage for a software vendor exhibiting 40% growth (VMware), NetApp was achieving fantastic growth.

That made me think, why isn’t NetApp being as highly valued as 3PAR by the market? I mean they are growing faster and have a more complete product offering.  So, why aren’t they getting as high a multiple? Hmm…could it be because the market doesn’t believe NetApp’s VMware driven growth will continue, that it is vulnerable and will come under strong competition? I can confirm VMDK/VDI optimization is a strong driver for our OEM customers, so I do see optimization rather than interoperability becoming table stakes at the VMware game.

We welcome BlueArc as a new OEM partner. Their high performance NAS offering will soon be even higher value and a perfect solution for VMware environments. My prediction is BlueArc and their partners will nick away at NetApp’s market leadership by offering the highest performing primary dedupe solution, Albireo and gain share in the high growth VMware market. Good luck to Gus and the BlueArc team.

A few comments have been made regarding latency, most recently in a Search Storage piece from Dave Raffo, when comparisons were being made between compression and deduplication applied to primary storage. The concern expressed is that deduplication may be inflicting latency in the write or read process for primary storage. That may have been an astute observation a year ago but today its simply just not a valid observation!

At issue is the performance of primary storage which needs to be as efficient as possible since businesses are run on primary storage that houses transactional data for core applications such as finance, manufacturing, CRM, sales operations, and supply chain to mention just a few. For these applications any significant delay in data access is simply not tolerable because businesses run on them! With today’s high performance data optimization (deduplication) tools such as Permabit’s Albireo there is no latency on data access because Albireo is NOT IN THE READ PATH!  And for that matter Albireo is not in the write path for all but in inline implementations.

In a recent blog post NOT in the READ Path I go into some of the details ‘As we designed and developed Albireo, we specifically addressed the most critical of all performance criteria - data read. Why is it significant? Because data is usually written once, but can be read hundreds, thousands or millions of times clearly affecting the performance of the storage.  As a result, Albireo is NOT in the read path — so there is no performance impact on read and no latency!

In the write process Albireo’s flexibility enables inline, parallel and post-process approaches (or any combination of the three) so the storage vendor can choose which approach they prefer. Let’s look at each of these:

* Of course in an inline approach, there is a very small write latency as Albireo does a hash key create and look-up. When we say very small, we’re talking a few microseconds. Can a user detect a few microsecond delay on write?  I think not!

* In a parallel approach. there is NO performance impact on write at all because the vendor continues to write the data as they normally do.  Albireo provides duplicate advice to the storage and the vendor can apply it when cycles permit.

* In a post-process deduplication approach, there is NO performance write impact because a write is done as the vendor normally would and a post-process deduplication can be initiated as cycles are available.’

Albireo is a new approach to data deduplication that is embedded into the storage stack using patented indexing, memory utilization and hash key technologies enabling scale out and performance that until Albireo has not been seen in other deduplication implementations.  Previous iterations of dedupe, which have primarily been used for backup, had their idiosyncrasies such as performance, latency, scale-out and memory usage to name but a few. These do not apply in Albireo’s implementation because we have solved these issues as we developed Albireo.

In summary, using Albireo High Performance Data Optimization (deduplication) in primary storage does not introduce latency!

Deduplication systems work by identifying and eliminating duplicate chunks of data. With Albireo, deduplication gets implemented by the OEM at the shared-storage, logical volume or filesystem level - the larger the shared pool of data on the back-end, the greater the potential for savings from redundant data. Albireo’s flexible deployment options enable deduplication to occur inline, parallel or as a post-process event to address varying performance requirements.  We’re working with some OEMs who are implementing all three options on the same systems.  Regardless of where or when deduplication is implemented, the process is the same; duplicate chunks are eliminated and data pointers are modified to share a single data chunk on disk.   Albireo’s job is to identify those duplicates as quickly as possible, while utilizing as few system resources as possible!

To help simplify the management of consolidated environments, VMware server virtualization allows administrators to create ‘template operating environments’, each with a standardized operating system and application environment. These templates are then ‘cloned’ into separate VMware images and installed as ‘guests’ on a physical server. The result is savings from simplified configuration management. Today, with modern processors from AMD and Intel, I’m seeing VMware users running modern server environments with 12 VM guests per physical server. I have heard of desktop virtualization environments with over 60 ‘guests’ per server. While VMware substantially reduced server costs, it has done nothing to help storage administrators who are left with the same number of operating system images on what is now a complex mixture of shared-storage resources.

It’s important to realize that while VMware provides a valuable cost benefit from consolidating IT servers, it offers no means for consolidating the storage used by VMware ‘clones‘.  That’s where deduplication comes in. You see it turns out that each of these ‘cloned’ VMware images starts off by taking up as much space as the template from which it was created. The set of data chunks making up a ‘template‘ and each of its ‘clones’ are nearly identical. When Albireo see’s these duplicate ‘chunks’ it identifies them and allows the storage system to replace them with pointers to the original data. Its easy to see why in a 12 VM guest environment, there’s nearly a 12:1 savings even after accounting for personalization of each guest. For a 60 VM Desktop environment, there’s a potential for almost 60:1 savings!

Next week I’ll talk a little about how Albireo can actually help VMware administrators better meet their storage performance requirements.

Albireo addresses the data safety issue head on. It never changes the data! Unlike a point solution or appliance, Albireo also doesn’t change the data or impact the read/write process (see NOT in the Read Path) because it is an advisory service to the storage stack. Simply said, Albireo analyzes the data, determines if it is duplicate and advises the storage that it is or is not a duplicate. If it is not a duplicate, the storage system does nothing to the data. If it is a duplicate, the storage system can create a pointer to the original data and remove the duplicate releasing storage space and reducing associated costs. The effect is that there will be less storage consumed, less power/cooling needed, less data center footprint and less storage to be bought to meet anticipated data growth.  These impacts drop to your CAPEX and OPEX providing budget relief!

How does this impact data safety? Simple, the data is written as it always would be by the storage system in its native. It is never changed or modified nor does it need an interpreter to reconstitute it.  There is no third party dependency! So if the data optimization functionality is disabled, removed, or fails data can still be read. Your data is as safe as it always is and is protected by your storage system!

Our partners will be deploying it in storage systems later this year and next year. If your storage vendor is not discussion this technology advance with you ask them about it!

Albireo is a technological step forward. It will impact your business in a very positive way and it will be easier, more efficient, eliminate risk and be less costly as it optimizes the data storage your business runs upon. Isn’t that what progress is all about!

‘Don’t impact performance’ I addressed in my blog last week: ‘Not in The Read Path’. Simply said, we don’t impact performance and provide a simple and elegant method of implementing deduplication. As a result of our approach, we developed a dedupe advisory approach that also enables the OEM to utilize all of their existing features as well!

When it comes to flexibility, we really took a step forward. We knew the concerns about deduplication technology revolved around 3 key areas:

* Hash key development/indexing and performance impact

* Resource utilization

* Deployment flexibility.

Hash keys are the lifeblood of deduplication in that they are the unique identifier of a data chunk. Once the key is developed for a chunk of data the next question is ‘have I seen this data before?’ This is where it gets difficult because doing this look up is what causes most dedupe systems to come to a grinding halt. Big indexes and large keys (yes we use SHA 256 so the key is large) create performance slow downs and in some lesser systems, scalability limits.  We skinned that cat with patented indexing that can return an index lookup in a few micro seconds! Hardly impact the performance at all! And scale out is also managed by patented memory based indexing that is 99.5% memory resident! No slow down there either!  Hardly ever see a disk fetch!   Once we do this we return advice to the OEM storage stack that the data chunk is a duplicate. The OEM then creates a pointer for the duplicate.

Resource utilization or how much of the processing power is consumed is a significant issue for storage vendors.  For example, newer systems have fast quad core processors (sometimes 2!)  while older or less high performance systems may have dual core processors.  In any case a flexible dedupe engine should enable the OEM to define the number of processors they used for Albireo depending on their overall system performance criteria. We designed that flexibility into Albireo to ensure our deduplication did not consume resources that would impact our partners performance.

Flexibility as you can see from the resource utilization comments is already there but we took an additional step by enabling our OEM partners to deploy Albireo in inline, post-process or parallel (hybrid) methods.  For example, if the partner performance tests enable an inline deduplication within the OEMs performance characteristics then they can implement Albireo inline!   If they want to balance their performance and deduplication efficiency then parallel mode may be the best for that partner. And finally, if high performance storage were the dominant criteria, then post-process mode for Albireo would work for that partner! We also give the option for the OEM to leverage all 3 so depending on use cases, they can implement the appropriate method (all within one system). Their choice!

All this is done with an SDK that can be integrated with 6-8 API calls. Some of our partners have integrated in as little as a few days. Albireo is an OEM partners tool that they can deploy based on their performance, resource and efficiency design criteria.  We made it that way!

In my next posting I’ll discuss data protection.

The most significant concern raised as we discussed Albireo with our partners has been the potential impact to performance. In many ways it’s their stock and trade and they protect it as any business should ‘with their lives!’ The thought of embedding deduplication technology, even though there is a clear customer benefit in cost, space and power/cooling reduction, is orthogonal to their performance mantra particularly in primary storage.

As we designed and developed Albireo, we specifically addressed the most critical of all performance criteria - data read. Why is it significant? Because data is usually written once, but can be read hundreds, thousands or millions of times clearly affecting the performance of the storage.  As a result, Albireo is NOT in the read path — so there is no performance impact on read!

In the write process, we also were careful to provide our partners choices that enable them to determine whether to impact their performance or not.  Albireo’s flexibility enables inline, parallel and post-process approaches (or any combination of the three) so the storage vendor can choose which approach they prefer. Let’s look at each of these:

* Of course in an inline approach. there is a very small write latency as Albireo does a hash key create and lookup. When we say very small, we’re talking mere microseconds across the entire data stream. Can a user detect a microsecond delay?  I think not!

* In a parallel approach. there is NO performance impact on write at all because the vendor continues to write the data as they normally do.  Albireo provides duplicate advice to the storage and the vendor can apply it when cycles permit.

* In a post-process deduplication approach. there is NO performance write impact because a write is done as the vendor normally would and a post-process deduplication can be initiated as cycles are available.

In summary, Albireo does NOT impact performance on read! There is also NO impact to write performance in parallel or post-process modes.  So, the vendor concerns about performance are addressed!

In my next post I’ll address implementation simplicity.

Usually summer is a time for reflection and sun tanning at the beach. From a business perspective things tend to slow down a bit while each of us takes a second breath and spend some quality time with our families away from the business side of our lives. There is usually not much industry news few ‘big deals’ occur that were not already in play. Technology shift announcements are typically saved for September when everyone is back, so there is as much traction as possible. Not the case for the summer of 2010!

It may have all started in early June when Permabit announced Albireo. Albireo is a simple and elegant data optimization technology that makes it possible to embed data deduplication into primary storage arrays with a few API calls to the existing storage firmware/operating environment.  The press, analysts, OEMs and bloggers,  were very supportive and effusive: ‘the tectonic plates of storage have shifted’ or ‘Killer dedupe’ or ‘This stuff is so far ahead in its capabilities and performance I can’t see why you (OEMS) would want to do it yourself’ to mention a few.  And yes, there were competitors snipes in the blogosphere - few were given any credence because of the support from press, analysts and OEMs. This started a domino effect that continues today.

A few weeks later, HP announced StoreOnce at their conference and talked about end-to-end deduplication (data optimization) setting a great vision! They went on to announce and describe their implementation of HP Labs deduplication technology (StoreOnce) via their D2D backup appliance. The rest of the details will evolve as they deliver and fulfill the end-to-end vision over the next 18-24 months.

Not to be upstaged by either a startup from Cambridge or the second largest storage vendor on the planet, EMC said ‘We got that’ as they announced their ‘Viper Team’ was working on data optimization (deduplication & compression) technologies that would concatenate the IP within EMC and offer an ability to apply end-to-end data optimization on their entire storage offering - another vision of the end state.  Details to be announced later!

Rumors were flying by then involving IBM and Dell acquisitions. . Dell would eventually buy Ocarina for their IP in compression and deduplication via Kleiner-Perkins with the implied intent to deploy data optimization across their storage offering. They also want to become a storage supplier beyond their own offerings by hoping to engage with the OEM partners that Ocarina had partnered with.

IBM would announce their acquisition of Storwize whose compression IP could eventually be embedded into IBM storage. This would be even more powerful if combined with deduplication (isn’t that data optimization too!) and reduce storage consumption for their customers.

All that in June and July!  What will August bring?

For end customers ‘this changes everything!’  Why you might ask?  Data stored in primary data stores will be compressed and/or deduplicated saving 50-75% or more of the storage space required. As a result, effective costs of storage will be substantially reduced, data center floor space requirements will drop proportionally, cooling and power will do the same and resources to manage ever expanding data stores will become more efficient. In parallel, storage drives will continue to become denser and 2, 4 and 6TB drives will be seen in storage array deployments during 2010 and 2011. Storage vendors will also begin deployment of more efficient unified storage technologies that incorporate embedded deduplication and/or compression, while legacy storage may be enhanced to incorporate embedded deduplication and/or compression. As end-to-end data optimization (extending the impact of dedupe/compression to secondary, archive and backup storage) is deployed, the cost reductions mention above regarding primary storage will multiply and storage expenses will drop significantly as a proportion of IT budgets! Bottom line - your CAPEX and OPEX will be significantly reduced as a result. That’s why the events of the summer of 2010 changed everything!

If your storage supplier is not telling you they have or will have data optimization in their primary storage offerings, if not end-to-end, I’d ask why not and reconsider your storage vendor choices!

And, yesterday, IBM announced the acquisition of Storwize. This is another great proof point for primary data optimization going main stream. Here’s how we view these developments:

• Primary data optimization is emerging and will gain momentum over the coming months as storage companies seek to protect margins and improve their competitive positions.
• Optimization technologies that are out of the data path and preserve data safety and storage performance will win in the market.

From our perspective, it’s still early in the game. Both Storwize and Ocarina sold data path appliances - and that is a very challenging (some would say, impossible) business model. Both are better as “owned” assets. Here is our take on the deals.

Dell/Ocarina

We are hearing from many who are questioning this transaction.

• Ocarina’s attempt to move from vertical (Media/Entertainment & Oil & Gas) to horizontal enterprise application will be a significant challenge. Moving from secondary and application niches to primary and enterprise - is like going from Summer League to the Big Show (baseball metaphor).
• Ocarina signed some nice names for “meet in the market” appliance deals, but received limited to no traction since their signing.
• Dell acquired an appliance technology (and maybe some beta code)? Is this a technology fit for Dell?
  • Dell has a limited storage portfolio today - they only own two SKUs of any significance (Windows Storage Server and EqualLogic).
  • Ocarina is a doubtful bolt on to CLARiiON which is in the middle of the Dell storage portfolio! How does Dell go end to end? Big stretch.
  • In this case end-to-end data optimization is at best point-to-point if the Ocarina embedded technology is valid!
  • EqualLogic begs the question of “what is the technology value of Ocarina content aware compression in block environments?”
  • Dell has had limited integration track record. Will Ocarina be different?
• Dell has some heavy lifting to do. End to end in anything is hard work and not for the faint of heart. It takes a full product portfolio and a robust technology solution to make it happen.

IBM/Storwize

From our perspective, IBM’s acquisition of Storwize makes sense.

• Compression is a great first step toward data optimization for IBM.
• If combined with a unified deduplication capability, IBM could have a best of breed data optimization solution giving IBM a lead on its competitors.
• Storwize compression today is an “in the data path” operation. From our perspective, that reason alone makes sense for IBM to own it - lock, stock and barrel - so it can be embedded into their storage architectures to minimize its impact.
• From our take, IBM hits a double (another baseball metaphor) and the heat will continue to quickly rise around the value of data optimization. How they move this forward to potentially score the winning run (yet another…) to compete with HP, EMC, HDS and now Dell, will play out over the coming months.

Permabit

There’s been much speculation around who Permabit is working with and what happens next.  We began introducing OEMs to Albireo primary data optimization about 9 months ago. We’ll confirm, we are working to enable dozens of storage products with deduplication that will enter the market in 2010-2012 and leave the speculation to Dave Raffo at SearchStorage and others as to who our partners are.  Again, it is early, but we are on a mission to provide Albireo data optimization to storage vendors to leverage their R&D and market reach and make their storage better, faster and more efficient.

We firmly believe for any company to make it past the early innings (final baseball metaphor) with an OEM data optimization solution, they need to offer the following:

• Solid, tested architecture - seasoned
• No performance impact - capable of handling demanding primary workloads
• No data safety compromise - out of read path
• Scale - to PBs of data capacity
• Best of class dedupe rates - for massive storage savings

Albireo offers all of that and the fastest route to market for storage companies looking to add data optimization to their products. For those of you keeping score, what do you think?