We have two announcements at the show. The first is around Permabit extending our VMware Alliance to the Select level.  That means we are partnering more closely with VMware and their ecosystem. We are the only OEM data optimization partner of VMware.  Our second announcement is of our new OEM relationship with BlueArc. Interestingly, the VMworld theme of every storage vendor, save two, is about ‘interoperability’. While good, interoperability or integrating with the VM management console isn’t that exciting to me. The real value is in optimizing the storage of VM. That brings me back to my main point - only two storage companies are talking about optimizing VMware environments - NetApp and our new OEM partner BlueArc and that sets them apart from the rest of the storage crowd.

A couple of weeks ago the subject of how important data optimization is to NetApp’s growth came up during a discussion with a prominent analyst. He attributed NetApp’s growth to two factors: adopting primary data optimization and hitching their wagon to VMware. In effect he said, by implementing a dedupe solution and becoming the preferred storage for a software vendor exhibiting 40% growth (VMware), NetApp was achieving fantastic growth.

That made me think, why isn’t NetApp being as highly valued as 3PAR by the market? I mean they are growing faster and have a more complete product offering.  So, why aren’t they getting as high a multiple? Hmm…could it be because the market doesn’t believe NetApp’s VMware driven growth will continue, that it is vulnerable and will come under strong competition? I can confirm VMDK/VDI optimization is a strong driver for our OEM customers, so I do see optimization rather than interoperability becoming table stakes at the VMware game.

We welcome BlueArc as a new OEM partner. Their high performance NAS offering will soon be even higher value and a perfect solution for VMware environments. My prediction is BlueArc and their partners will nick away at NetApp’s market leadership by offering the highest performing primary dedupe solution, Albireo and gain share in the high growth VMware market. Good luck to Gus and the BlueArc team.

And, yesterday, IBM announced the acquisition of Storwize. This is another great proof point for primary data optimization going main stream. Here’s how we view these developments:

• Primary data optimization is emerging and will gain momentum over the coming months as storage companies seek to protect margins and improve their competitive positions.
• Optimization technologies that are out of the data path and preserve data safety and storage performance will win in the market.

From our perspective, it’s still early in the game. Both Storwize and Ocarina sold data path appliances - and that is a very challenging (some would say, impossible) business model. Both are better as “owned” assets. Here is our take on the deals.

Dell/Ocarina

We are hearing from many who are questioning this transaction.

• Ocarina’s attempt to move from vertical (Media/Entertainment & Oil & Gas) to horizontal enterprise application will be a significant challenge. Moving from secondary and application niches to primary and enterprise - is like going from Summer League to the Big Show (baseball metaphor).
• Ocarina signed some nice names for “meet in the market” appliance deals, but received limited to no traction since their signing.
• Dell acquired an appliance technology (and maybe some beta code)? Is this a technology fit for Dell?
  • Dell has a limited storage portfolio today - they only own two SKUs of any significance (Windows Storage Server and EqualLogic).
  • Ocarina is a doubtful bolt on to CLARiiON which is in the middle of the Dell storage portfolio! How does Dell go end to end? Big stretch.
  • In this case end-to-end data optimization is at best point-to-point if the Ocarina embedded technology is valid!
  • EqualLogic begs the question of “what is the technology value of Ocarina content aware compression in block environments?”
  • Dell has had limited integration track record. Will Ocarina be different?
• Dell has some heavy lifting to do. End to end in anything is hard work and not for the faint of heart. It takes a full product portfolio and a robust technology solution to make it happen.

IBM/Storwize

From our perspective, IBM’s acquisition of Storwize makes sense.

• Compression is a great first step toward data optimization for IBM.
• If combined with a unified deduplication capability, IBM could have a best of breed data optimization solution giving IBM a lead on its competitors.
• Storwize compression today is an “in the data path” operation. From our perspective, that reason alone makes sense for IBM to own it - lock, stock and barrel - so it can be embedded into their storage architectures to minimize its impact.
• From our take, IBM hits a double (another baseball metaphor) and the heat will continue to quickly rise around the value of data optimization. How they move this forward to potentially score the winning run (yet another…) to compete with HP, EMC, HDS and now Dell, will play out over the coming months.

Permabit

There’s been much speculation around who Permabit is working with and what happens next.  We began introducing OEMs to Albireo primary data optimization about 9 months ago. We’ll confirm, we are working to enable dozens of storage products with deduplication that will enter the market in 2010-2012 and leave the speculation to Dave Raffo at SearchStorage and others as to who our partners are.  Again, it is early, but we are on a mission to provide Albireo data optimization to storage vendors to leverage their R&D and market reach and make their storage better, faster and more efficient.

We firmly believe for any company to make it past the early innings (final baseball metaphor) with an OEM data optimization solution, they need to offer the following:

• Solid, tested architecture - seasoned
• No performance impact - capable of handling demanding primary workloads
• No data safety compromise - out of read path
• Scale - to PBs of data capacity
• Best of class dedupe rates - for massive storage savings

Albireo offers all of that and the fastest route to market for storage companies looking to add data optimization to their products. For those of you keeping score, what do you think?

Economics

Usually around election time the old theme ‘it’s the Economy stupid’ gets hauled out. Well, the storage sector is a very competitive market and when economics are at stake - meaning P&L, earnings per share, gross margin, revenue and market share - things happen fast.

I’ll keep this short and to the point. Let’s look at two storage companies, ABC and DEF.  ABC is the industry leader. They sell $billions of primary storage at $10/GB and they operate at a 60% Gross Margin. Their competitor, DEF is the price leader and they sell $billions of storage at $8/GB and operates at a 50% Gross Margin. Their product P&Ls look like this:

Now, ABC adopts data optimization. Let’s assume they keep their prices the same, but deduplication enables them to lower the effective cost of storage by 60% passing on huge savings to their customers and gaining market advantage over their competitor. DEF does not adopt data optimization. The customers of ABC and DEF see effective cost of $4/GB and $8/GB, respectively.  Wow!   ABC is now the price leader and aggressively takes market share from DEF.

DEF can react one of two ways. They can cut prices, but they are already at a Gross Margin disadvantage or they too can add data optimization into their solutions.  It’s clear that the latter option is where the industry is headed.

I said I would keep it simple.  Economics - margin dollars and market share — are the reason virtually all storage vendors are moving to data optimization and why they are doing so at a fast pace.  Customers become the ultimate winners as they see their effective cost of storage dropping dramatically along with falling data center and storage management costs.  To storage companies ABC and DEF, as Senator Dirksen said, ‘a billion $ here and a billion $ there, pretty soon you are talking about real money’.

That shocked me. So I turned off the Sox game, sacrificed a couple hours of sleep and dug into the ultra comprehensive, Gartner External Controller-Based Disk Storage, Worldwide, 2006-2009 report to see how we are doing in the storage business (another wild night at the Cook household).  The report shows the top five US based storage companies (EMC, IBM, HP, Dell and NetApp) maintained worldwide market share between 67-68.9 percent during the period and companies such as 3PAR, DataDirect, DataDomain, Xiotech, Compellent, Pillar, BlueArc, Isilon,  LeftHand and EqualLogic (all grown in the US) achieved noteworthy market share during the period. Not one non-US based company entered the top 25 storage companies during the period according to Gartner.

My (slightly) unscientific analysis is:

the US storage industry is competing well and doing a great job at innovation. It is interesting to think about how it is being done. The storage industry is innovating the old fashioned way: with internal development, OEM licensing and strategic acquisitions. By the way, I’m not saying innovation is unique to the US, XIV is a great example of a company with origins outside the US (Israel) that was acquired by IBM.

Think about recent announcements in data optimization. During EMC World in June, the company announced data compression and on June 14, IBM’s potential acquisition of Storewize was reported.  On June 7, we announced Albireo OEM deduplication to the market and during HP’s Tech World the announcement of StoreOnce end-to-end dedupe was made followed by an EMC announcement of dedupe skunk works, Viper. So, in the course of just one month, we’ve seen internal development , OEM deals and strategic acquisitions all employed by leading companies to make their storage more efficient.  The market works.  Innovation is fueling rapid adoption of data optimization.

Here’s why. Data optimization will put late adopters at a pricing disadvantage. By lowering the effective cost of storage by 60-80%, data optimization is a must have technology or a vendor will face margin pressure and market share decline as competitive adoption takes hold.

When we invented Albireo, we noted a lot of great companies were building outstanding primary storage solutions. We concluded the last thing the world needed was another storage solution.  Albireo was designed to make their primary storage solutions better (NAS, SAN and Unified) - massively more efficient without performance impact, feature loss or data safety compromise. OEMs told us, ’stay out of the read path and use my storage stack’ and we did. Albireo leverages the vendors’ data management IP and is never in the read path so their customers can sleep at night. And it accelerates time to market so they can weigh the build vs. OEM decision and sleep at night. As Steve Duplessie said, ‘ This stuff is so far ahead in its capabilities and performance I can’t see why you would want to do it yourself, unless you already have it baked.‘

From what I see, healthy competition is driving aggressive innovation and adoption of data optimization technology by primary storage vendors. Market forces are working and there is no ‘crisis of innovation’ in this market. What do you think?

First let’s look at the basics of compression.

Compression works only in file based storage compressing each file, but does not function across files.

Compression identifies redundant data across a very small window, usually 64 KB.

Compression produces data reduction rates at most 2X for most data types.

Compression alters the underlying data structures and requires compression and decompression of data.

Compression operates in the data path and impacts read/write performance as a ‘bump in the wire’ (kudos to Storwize for their work to improve performance).

Compression is a potential single point of failure for data retrieval.

Given these attributes, compression can provide a level of data optimization for NAS systems, but the only safe way to implement compression is as an embedded feature in the storage software - deployed, owned and managed by the storage vendor (not OEMed or deployed as a third-party stand-alone appliance in the read/write path). Embedded compression is starting to take hold with NAS OEMs (EMC and IBM speculation) and we expect to see widespread future adoption. Again, Albireo works well with compression, so we support this incremental data optimization move.

And now let’s look at Albireo embedded data deduplication. (See Jered Floyd’s detailed blog on the key attributes for a high performing data optimization solution here.) Here’s how Albireo stacks up:

Albireo supports block, file and unified storage architectures.

Albireo dedupes data across the entire storage pool, up to petabytes of data.

Albireo produces 3.75-100X data reduction for typical enterprise data types.

Albireo doesn’t alter the underlying data structures.

Albireo operates out of the data path with no impact to read and write performance.

Albireo operates as an inline, parallel or post-process operation and is never a failure point for the storage system.

So let’s summarize the comparison of Albireo embedded data dedupe and compression technologies in terms of data safety and business value.

Albireo deduplication outperforms compression for data reduction by an order of magnitude and insures enterprise class data safety. Superior business value and data safety - that is Albireo data optimization for primary storage.

In my next post, I’ll discuss how combining Albireo embedded dedupe and traditional compression provides best of class data optimization.

My daughter and I sacrificed no performance reduction (0-60 mph, braking, driving range), no safety rating decline (not the size of a postage stamp, solid vehicle) and no feature hit (comfort, curb appeal, capacity, cooling, sound, guidance, sunroof) to drive efficient and clean vehicles.

New technologies like these take broad hold when you sacrifice nothing to attain efficiency. My daughter and I can safely and comfortably drive in the left lane, and we can do it in our fuel efficient and clean vehicles.

I believe there are many parallels to primary data optimization adoption.  We all want it.  We all need it.  If deduped primary storage was as fast, as feature rich and as safe as non-optimized storage we would have it already.  Let’s examine each of these key adoption hurdles and how they can be addressed in data optimized primary storage.

Maintain High Performance - Customers demand storage performance.  They have built their entire production environments and workflows around high I/O rates.  And large primary storage companies like EMC, HDS and HP and smaller high performance NAS companies like BlueArc, Isilon and Symantec have invested hundreds of millions in R&D focused on performance to enable customers to process and store PBs of data at near-line speeds.  I/O speed, like 0-60 acceleration is here to stay.  So, to be broadly adopted, primary dedupe cannot impact I/O.

Here are two ways to solve this challenge.  First, data optimization cannot be a “bump in the wire”, it must be out of the data path. Second, it must have the flexibility to be deployable as a parallel or post-process operation depending on the workload, platform, and required performance characteristics of the storage.

Preserve Features - We like our bells and whistles.  Storage providers have made large investments to differentiate their primary offerings, each with their own flavor of snap shots, replication, provisioning and many other features.  Once an enterprise adopts these features, they won’t give them up.  However, inline deduplication can mask many of these features and/or can render them inoperable.

Again, the answer is to deploy dedupe out of the data path.  When deployed out of the data path, deduplication optimizes many features and also shrinks the storage footprint.  So, you can have the sunroof, heated seats and full size vehicle as well as fuel efficiency.

Assure Data Safety - Customers assume primary storage will protect their data and, again, storage companies have made huge R&D investments to protect information integrity. Just as I wasn’t willing to have my daughter drive an unsafe, “toy car” to achieve fuel efficiency, data safety in a deduped system cannot be compromised.  Any “bump in the wire” solution introduces a single point of failure to the storage system.  But wait a minute, it gets worse.  Many solutions are not only in the data path, but they also alter the data structures.  Enterprises must run from any solution that, “hydrates or re-hydrates” data.  When data is altered, it is inaccessible if the application is ever unavailable or down. The point is simple, in a deduped primary system, the data must remain unaltered and data safety must be managed by the storage stack - any less is a step backwards in data protection.

Up to now, adopting primary deduplication took primary storage from the left lane to the break down lane.  Today, I’m pleased to announce our revolutionary OEM Data Optimization SDK, Albireo.  Permabit is delivering Albireo to storage OEMs who will bring it to market in their products beginning later this year. Albireo is the result of nearly ten years of R&D focused on advancing deduplication technologies.  When our CTO (and founder), Jered Floyd conceived of Albireo it was with a singular mission to deliver data optimization for primary storage with no performance reduction, no data safety decline and no feature hit.  Albireo does all this and it leverages all standard storage architectures and produces best of class deduplication rates.

That is Albireo.  Data Optimization for Primary Storage - without compromise.

Embrace efficiency.

New information sources continue to produce mind boggling data growth. IDC predicts there will be 35 ZB (that’s zettabytes!) of data stored by 2020.  So we need to attack all cost components of storage.

Let’s look at two fundamental storage efficiency moves that will hit the market in force this year and become dominant trends by 2011: unified storage and data optimization.

Unified storage, standardized architectures or converged platforms - whatever you call them - are all about driving cost out of storage hardware.  Products coming to market soon will combine SAN, NAS and Object based storage software using a common hardware platform.  Think about it, this is the same cost efficiency move that server companies made by moving to standard components!  And, more recently, similar to what Google, Amazon and other companies have achieved by deploying software innovation on top of highly redundant and inexpensive hardware.

This is a massive development undertaking to ‘unify’ their block and file storage offerings and the long-term benefits are huge.  First, they will achieve cost efficiency and scale economies by supporting one common platform, rather than three or more. And, they will get out of the hardware platform development business!  Second, and more importantly, they will achieve greater software development efficiency by deploying software across all of their platforms. In fact, R&D will be focused on delivering common software attributes that span the unified offering and leverage the block or file system capabilities while taking advantage of the multi-processor technology on their next generation platforms.  Future software development efforts will deliver a software layer that delivers features over SAN, NAS and object storage.  That is software development leveraged over a larger business base and that equals R&D efficiency.

One high-impact software solution that is becoming ‘table stakes’ by customers is data optimization - eliminating duplicate blocks or chunks of data and 50% or more of the storage footprint. In order to be a unified feature, the data optimization solution must globally address both block and file storage as a software layer that handles deduplication  - remember this is a unified storage solution.  One deduplication scheme for the block system and another for the file system data isn’t a unified software solution.  I’ll get to other required attributes for unified storage data optimization in my next post, but for now let’s just say the unified software must leverage the huge development investment made in the unified platform.

I think the killer unified storage offerings will deliver massive storage efficiency with the one-two punch of unified platform and data optimization software in one comprehensive solution.  That will be a challenge for many storage organizations. Both unified storage and data optimization promise to be table stakes for next generation storage and will be requirements in the battle among major storage vendors to protect margins and drive to expand market share.

You might ask why a guy who is negative about storage spending in general is leading an expansion stage storage company. Well, I’ll skip to the punch line and tell you I am optimistic about value creation via storage optimization technologies.

When the economy took a turn in 2008, I believe everything changed in storage.  Budget constraints shined the light on storage efficiency in a way the industry had never experienced. That meant the focus in storage innovation changed from speed (frankly, we can store and access information plenty quickly these days) to cost optimization. Simultaneously, the largest storage vendors got conservative and slashed R&D budgets to build cash balances (and they weren’t alone, the US tech sector has unprecedented cash balances today). And, they focused on maintaining account control by slashing prices on less efficient technologies.

By cutting back on investment before and during the down-turn, the largest storage vendors missed a cycle of innovation. Their product portfolios are behind in the areas of deduplication, scale-out technologies, unified storage and automatic tiering/storage virtualization technologies. Now they need to catch up.

Well capitalized, up and coming storage companies continued investment during the downturn as innovation is their only competitive advantage and their place in the ecosystem.  If they invested in storage optimization and proved their technologies, they are more valuable in today’s world where the key driver for storage adoption is cost efficiency and mature technologies are scarce.

So, what do I conclude this means strategically for storage vendors?  First, lets talk about the leading storage suppliers and then the up and comers or innovators. I should note my simplistic view of the storage world is one in which the leading vendors sell 95% of storage and account for 5% of the innovation, while up and coming companies sell 5% of the storage but account for 95% of storage innovation.

For the largest storage vendors, it is time to open up the spigots and drive investment for storage efficiency.  Someone way smarter than me said, “you can’t save your way to prosperity”.  Now is the time for storage leaders to aggressively invest in dedupe, scale-out, storage virtualization and unified solutions.  They must buy, license or build optimization technologies with a mind set of “eat lunch or be lunch” because their competition is. We’ve seen the start of this with recent M&A activity. Stop debating “is dedupe ready for primary” or “my solution is faster than X” and “SATA vs. fiber channel”.  The facts are: dedupe is ready for prime time today and is coming like a freight train, speed is assumed and in the big picture media form factor is irrelevant. Get on the optimization bus or risk market share erosion.

For the up and coming storage innovators, focus on transformational technologies that can massively lower the cost of storage for IT organizations. If you aren’t focused on optimization technologies and cannot reposition, return your investors money. Build your optimization IP portfolio and work with leading storage vendors (OEM) who will leverage your technologies - remember they have 95% market share!

I see a transformational change occurring in storage efficiency driven by aggressive M&A and investment activity by leading storage companies and resulting in massive cost savings for IT organizations. Leading a company that is driving key technologies for the transformation is exciting stuff.  I love this job.

1. Drive-up storage utilization. By eliminating redundant processes and inefficient LUN allocation the typical enterprise can delay storage expansion by 12-18 months. Start with an audit of your storage practices that includes an application by application analysis of redundant data copies generated and systematically eliminate them. Then evaluate utilization and fill rates by LUN. Use this as a guide to “free-up” space.
2. Align storage performance with required service levels. Over time, “primary” storage has become the largest part of many enterprise storage environments. This is inefficient as “primary” storage is purpose built to address only transactional data. Return Tier 1 storage to its proper place and aggressively move non-transactional data to cost effective near line or value tier storage and realize 10X-50X cost savings.
3. When evaluating additional storage investment or issuing an RFP, make sure your storage purchase has the flowing attributes:
   1. Thin or zero provisioning to eliminate inefficient LUN allocations and drive up utilization.
   2. Integrated Dedupe/Compression space reduction. Your savings should be proven by the vendor prior to purchase for ROI calculation.
   3. Incorporates high-capacity (1TB+) disk drives or SSDs. To achieve improved performance to power ratio and decrease footprint.
   4. Incorporates modern processors. Again to achieve improved cost and performance to power ratio.
   5. Scale out architecture. Your information needs are growing. The storage solution must provide one management interface for hundreds of TBs to PBs of data.
4. Add expansion stage storage vendors to your approved vendor list. Gartner recommends selectively including innovative vendors in your environment to take advantage of advancing technologies and to keep entrenched vendors competitive.
   1. Best of Breed vendor selection has never made more sense.
5. Before your next storage purchase, engage with your vendors and have them present their product roadmap to you. Make sure your current purchase is with a company that embraces cost efficient storage technologies in their product plans.
6. Finally, never purchase more storage than you need. It will cost less next quarter and next year. Your scale-out vendor should supply incremental additional capacity at a lower cost as required.

The “New Normal” in storage is a great opportunity for enterprise storage organizations. You can now meet the challenges of your rapidly growing storage needs within budgetary constraints.

PR: wait… I: wait… L: wait… LD: wait… I: wait… wait… Rank: wait… Traffic: wait… Price: wait… C: wait…

Below, I illustrate four scenarios for storage revenue/expenditure growth in the future.  Line A is the one we saw prior to 2008, where storage revenue followed an upward sloping function of capacity and price that equates to revenue increases.  That was the storage world over that last 10 year period (ex-some variability).  However, beginning in 2008, storage revenue began to fall (B) and diverge from the upward sloping line.  Respectively, lines (C), (D) and (E) represent future scenarios of (C) full recovery to pre-September 2008 pricing and demand, (D) recovery after reset and (E) a permanent decline and slower future growth.

I think we are entering a long-term period of slower revenue/expenditure growth for storage represented by (E).

Here’s why:

1. Buying behavior has forever changed.  We met with a large energy company recently.  They were discussing a Q1 2010 buy (that may slip a quarter or two), but mentioned they had eliminated 100% of storage purchases in 2009 - they only spent for maintenance and support.  By shutting down purchasing they forced their organization to push up utilization.  We have heard a lot of this recently.  Do the math, if on average an organization’s storage is 60% utilized (and information is growing at 40%), by simply saying “no” they can put off new purchases for 18 months and then if they eliminate the redundant information (See Hu Yoshida’s thoughtful post) and other inefficiencies they can push purchases off another 18 months.  That means many organizations don’t need more storage for 3 years. Buyers change behaviors reluctantly and slowly.  Call it inertia or elasticity, I think we are witnessing a permanent behavioral change to higher storage utilization and decreased consumption by enterprises.
2. Pricing behavior has changed.  Storage prices plummeted over the last year.  To counter the pull-back by enterprises, storage companies slashed pricing and sold at cost or below cost to gain deals and maintain footprint.  Customers are savvy and pricing elasticity is sticky downward (erosion is permanent).  In fact, Gartner predicts sub-$1/GB pricing in 2010 for NAS - this is in world where primary disk cost $40/GB just two years ago. For the 6 year period of 2008-2013, Gartner points to demand growth (CAGR) of 11% and pricing decline (CAGR) of -28% - pointing to a period of negative growth in the market.
3. Cost efficiency innovation is coming of age. Customers are looking for lower TCO and expansion stage storage vendors are supplying it.  Market share is being gained by companies who are leading the way with compression, deduplication (see dedupe2.com), thin or no provisioning/virtualization and clustering/scale out. Look at the following table.  The public companies that are leading the charge with cost efficient technologies are outspending entrenched players in R&D (as a percentage of revenue) and they are gaining share because they offer a compelling value proposition.




This is great for the enterprise buyer.  They now have more storage choices and are shifting purchases to the lower cost solutions.

4. Cloud shifts power to buyers.  By consolidating demand, Cloud providers can wring margin out of storage and assemble best of breed, guaranteed solutions for more businesses. This is great for enterprise buyers because they will get the best solutions and they can buy like the largest organizations to drive their storage costs down. This isn’t great news for storage companies, however.  They will have gained a new channel, but it will be concentrated and a very price sensitive one for sure.

So, what does this mean?  I conclude the effective price of storage will decrease almost as rapidly as demand grows.  It means the New “Normal” is about marginal growth in storage investment and much greater cost efficiency.

In my next two posts, I will talk about how the “New Normal” impacts the enterprise and storage companies and how they can adapt to prosper in the new environment.

PR: wait… I: wait… L: wait… LD: wait… I: wait… wait… Rank: wait… Traffic: wait… Price: wait… C: wait…