I’ll comment on his technical concerns.

Compression and deduplication are very similar in that they identify and eliminate redundant data, but the scope of this duplicate identification is vastly different. Traditional compression works on a small window of data and with short duplicate segments so that the compression tables fit efficiently in a very small amount of memory. Storwize may not be using a 64 KB window, but I imagine the order of magnitude is about right… and that’s not a criticism of their technology at all. In fact, the way Storwize manages data in chunks so that they can maintain performance is very clever.

Calling deduplication lossy is nonsense; both compression and dedupe replace redundant data with references to other instances of that data, just at different scales as I note above. Unlike Ocarina’s NFO, which frighteningly throws away actual content, both dedupe and traditional compression return the original bitstream.  Tom’s point was that Albireo embedded dedupe leverages existing file and block system concepts to make those references so no interaction with our software is required on read, while a compression appliance modifies the data format before it reaches the storage array, which creates data lock-in. Take away the appliance, and the storage is full of uninterpretable data. That’s a concern for storage vendors and users alike.

As to the chart, when you look at this as ‘embedded dedupe’ vs. ‘appliance-mediated compression’, you can see why Tom says that appliance compression alters the data, and Albireo dedupe does not require ‘rehydration’.  As for ‘optimizes block’, I haven’t yet seen Storwize’s block optimzation products, so I can’t comment, but I do wonder how they make the space saved to compression available to the user? We agree that savings are absolutely data dependent. In general, deduplication alone offers more savings than compression alone, and both together give the best results by far. Perhaps we can work together to ensure Albireo and Storwize yield optimal results?

First, Albireo supports block, file and also new unified or converged storage platforms.  By addressing all types of primary storage, we avoid leaving huge amounts of users’ data unoptimized.  Additionally, next generation storage platforms put block and file data on the same underlying storage, and Albireo makes it possible to identify and deduplicate data across both.

Next, we uniquely provide the ability to scale deduplication across a pool of storage many petabytes in size, instead of limiting deduplication to smaller islands of a few terabytes.  This is critical to delivering high rates of deduplication.

Further, Albireo delivers sub-file, content aware deduplication.  Whole file single instancing just doesn’t cut it for common primary data files, like Office documents or virtual system images. Albireo can identify optimal boundaries in a variety of file types and then deduplicate segments as small as the storage can support.  This delivers industry-leading deduplication efficiency to our partners.

As I explained in my last post, Albireo is successful because it is an embedded, integrated solution.  Integrating directly with primary storage vendor’s technology, it leverages all their existing R&D.  We also provide the capability for Albireo to be integrated as inline, post-process, or parallel deduplication, whichever matches the underlying storage platform the best.  This means that there are no rough edges where features of the underlying storage are lost; the deduplication is transparent and automatic.  Users may not even know that their storage is using Albireo for deduplication, except for the levels of savings and performance far beyond what anyone has seen before.

Finally, because Albireo is delivered as a software tool kit, it is integrated outside of the storage read path.  Our technology solves the hardest parts of deduplication, namely sub-file duplicate identification, and then leverages existing vendor file and block system metadata for eliminating duplicates.  Because of this, read operations only look at the file or block metadata without the need to consult our indexes, meaning we have no impact on performance, functionality, or data integrity.  Even if our software is turned off, all user data remains accessible.  This is completely unique in the industry.

I’m extremely excited to be able to now talk publicly about Albireo and the deduplication benefits it provides to existing primary storage technologies.  We’ve been focused on this for the past year, and the success of our partners’ integration efforts has confirmed the ease of integration and technological power of the Albireo toolkit.  Through our partners users will be using this soon, and I’m sure that they’ll be pleased with the results.

Permabit has been in the deduplication business since 2000, more than ten years, and we’ve learned a great deal about both technology and customer requirements. In fact, we’ve explored delivering deduplication to the OEM storage vendor market for some time, after including it for many years in our Enterprise Archive product. If you’re not familiar with it, our Enterprise Archive is a complete stack solution for efficient value-tier storage, delivering our own file interfaces, file system, RAIN-EC data protection, and hardware.  When talking with tier 1 storage vendors we were told many times, “we’ve invested millions in our file systems and data protection; yours is great, but we really just want deduplication.  Can you give us just that?”

For a long time I, along with the rest of the industry, thought the answer was “no”. When our engineers explored just providing dedupe we ended up with complex “bump-in-the-wire” appliances that sat in front of the storage and treated it almost like JBOD, masking performance and functionality, and jeopardizing integrity through data lock-in to the solution. We didn’t find this acceptable, and refused to try and sell it.  Others weren’t as resolute and have tried to bring solutions like this to market and have found it challenging and less than rewarding.

Then, a bit over a year ago, I had an idea. What if we could provide a development kit that delivered the core technologies in deduplication and integrated into the vendor’s existing storage stack, rather than sitting outside it? We could avoid competing with our partners on functionality, and eliminate the concerns that Tom explained so clearly. That’s what Albireo is - the core technologies that make fast, scalable deduplication possible, packaged in a way that they can be integrated into any storage vendor’s stack in a matter of days to weeks. Our engineering team took this idea, extracted several years of deduplication effort from our core research, enhanced it further and packaged it as a complete SDK, all in under a year.

I named this project Albireo, after the most visible binary star system. I thought this captured the basic idea of the product; a technology that works alongside an existing storage stack to deliver a powerful deduplicating storage solution that looks and acts as a single product. And of course, it also makes multiple data instances appear as one.

In the process of developing Albireo we learned seven key requirements for an integrated, deduplication solution. Be sure to read my next post to find out how these have all been critical to the success of Albireo with our partners.

Their biggest NAS competitor took the bait, and NetApp jumped into the fray. In between the vitrol some good points are made about why Chuck is wrong. For example, if deduplication is increasing access to common blocks it means that you’ll be seeing much better cache efficiency, which will offset additional load on the drives. The “boot storms” he talks about with many virtual machines hosted on the same storage are actually less likely to occur with deduplication than without!

Now Hu Yoshida has weighed in on Chuck’s side, but then laid out the much more reasonable view that virtualization, tiering and dynamic provisioning are critical in an environment with costly top-tier primary storage. That’s correct, but it doesn’t mean that deduplication at that top tier isn’t a huge win as well.

Deduplication has seen its first successes in the D2D backup space, where it’s easy to get a lot of deduplication due to the data patterns and traditional backup schedule. Applying deduplication beyond backup is hard, because the opportunities for deduplication are fewer and further between, and so these D2D backup devices have never been able to address archive or primary storage effectively. That doesn’t mean dedupe is bad for primary, it just means that it’s harder to do.

At Permabit, we consider dedupe for backup to be Dedupe 1.0, and the future for innovation is in Dedupe 2.0, which includes dedupe for primary and cloud storage. We host a forum over at Dedupe 2.0 to discuss this further, and recently released our Permabit Cloud Storage product to address these new customer needs. I can’t give too much detail, but we’re constantly at work making our deduplication technology available to ever broader markets.

Dedupe for primary is a huge win for the storage consumer, but it’s taken us nearly a decade of extensive technology and patent development to solve the scalability and speed challenges needed for that market. I think it’s no coincidence that the two voices denouncing primary dedupe the most, HDS and EMC, has no products to offer that include a feature which will soon become a customer requirement.

If you’re going to be at Storage Networking World next week and would like to hear more on primary dedupe, Arun Taneja is moderating a panel, “Primary Storage: The New Frontier for Data Deduplication“. I’ll be there, along with Val Bercovici from NetApp, Carter George from Ocarina, and Peter Smails from Storwize. It should be a lively discussion! Perhaps Chuck will stop by?

They can, but it depends a lot on when and how you encrypt your data. The challenge is in balancing when during your information lifecycle your data is encrypted, and how it is handled. If you encrypt data high up the stack, in the application, then it’s more likely to be end-to-end secure, but cannot be easily shared with other applications in your environment. If you encrypt data lower down the stack, in the storage, it can be easily shared but you must be careful how it is protected in transit. In many environments, different cryptographic implementations are appropriate for different kinds of data.

Broadly, there are two areas in which to consider encryption implementation — transport encryption and data encryption. Transport encryption is where all communication between two applications or servers is encrypted, delivering a secure communications channel. This means that both data and control commands are encrypted; this is important because it means that an eavesdropper cannot see your data and also cannot tell what sort of things you are doing with that data. With transport encryption, the data on the other side of the connection is generally handled in unencrypted form, so this does not protect against data leaks or maliciousness within the application. HTTPS is a common example of use of transport security.

Data encryption, on the other hand, is where the individual pieces of information being processed are encrypted. These may be processed by the application in an encrypted form, stored into a database, or written to disk. An untrusted application, like perhaps a storage system, can be handed data that has been encrypted without concern that the information will be leaked. An encrypted archive file on disk would be an example here. In some ways you could consider tape encryption as either or both types of security. The tape is carrying data and control information between two applications (or two runtime instances of the same application), so it could be considered a form of “transport” security, but if you consider the third-party handling your tapes as part of your storage infrastructure, it’s more like a form of data encryption.

When selecting a product that incorporates encryption one thing to consider is the encryption algorithm used; luckily, this is an easy choice. Only use products that implement AES, the Advanced Encryption Standard. Anything else is unlikely to be (or remain) secure…. the only possible exception would be Triple DES, but it is a cipher that is showing its age.

Recommendations for Deduplication

If you would like to combine deduplication and encryption, at some level the storage stack must have access to the unencrypted data so that it can identify duplicates. For a system with standard protocol interfaces, such as NFS and CIFS, this means not performing data encryption within your application on data that you want to deduplicate. This doesn’t mean not to use encryption at all, however.

Data that must be kept very secure and is also unlikely to deduplicate, such as credit card numbers, can be safely encrypted by the application. For protection of the remaining data, you must select a system that both delivers transport encryption between your application and the storage, and also includes data encryption internally to protect data on disk. Surprisingly, there aren’t many options available today.

Several drive vendors have begun to incorporate full-disk encryption (FDE) into their disks, and storage array vendors are just beginning to make use of this. This means that the data on such drives is protected against theft or loss of the drives, but the weakness is still that the device the drives are in must have the keys to unlock them. That means if someone walks away with a server and its disks, all bets are off. FDE drives are compatible with deduplication, though, because any deduplication activities are happening at a higher level in the storage system.

Permabit Enterprise Archive supports encryption at a number of different layers. This includes encryption to the client, encryption during replication, and encryption on disk. Permabit always uses the AES cryptographic algorithm, as mentioned above.

First and foremost, transport encryption is used wherever possible. If the application protocol (i.e. NFS, CIFS) supports an encrypted connection, we will deliver that. Unfortunately, this is not widely available today, with CIFS supporting secure authentication and some recent versions supporting secure transport.

Replication is always performed over an encrypted channel as well, even if the data being transported is already encrypted. This ensures that customer data is not replicated to an attacker outside your firewall that has surreptitiously tried to intercept your replication data stream. Additionally, because the transport channel is encrypted, an eavesdropper cannot tell anything about the sort of data being replicated, such as file sizes.

Finally, Enterprise Archive can optionally encrypt data on disk so that it is protected against theft or loss of the hardware. This option can be configured on a volume by volume basis. If someone were to walk away with one (or even all) of the hard drives in an Enterprise Archive install, they would not be able to make sense of any of the data. This offers strong protection against data theft from the equipment.

Permabit’s on-disk encryption offers additional protection beyond what full-disk hardware encryption offers, because of how encryption keys are handled. In a system with FDE disks, encryption keys must be stored on the server so as to unlock the disks when they are powered on. While this means that a stolen disk is of no use, a stolen server necessarily contains the keys that will unlock its disks. In Enterprise Archive, data encryption happens at the access node layer, significantly reducing the vulnerability profile of the keys. Because application transports like NFS require cleartext data, the access nodes must have access to the data encryption keys. For encrypted volumes, they encrypt and decrypt data as it flows from and to the storage nodes in the system. This means that storage nodes never have the keys necessary to decrypt the data that they hold and protect, a significant separation of responsibility.

Overall, deduplication and encryption are compatible, but to use them together you must take care on where you apply the encryption. For data to be deduplicated, encryption must take place within the storage system, and not at an application or gateway layer. To ensure data security, make sure information is always encrypted in flight whenever possible, especially during replication or backup.

Disk-based backup is a huge market, don’t get me wrong, but it’s tiny compared with the amount of disk out there for archive and primary storage. Not only that, but as I wrote about in “Cutting Costs with Enterprise Archive”, deduplication for disk-based backup is based on “doing it wrong” — intentionally writing the same data again and again even though there’s no good reason for having configured your backup software to do so. Dedupe for backup was always a marketing game, one that Data Domain’s team excelled at winning.

You might note that we’ve never pursued this crowded D2D market head-on; instead we’ve focussed on where more data lives and nobody else has done a good job at addressing the much harder technical problems: Dedupe 2.0, archive and primary storage. If you’ve been to our website recently, you may have noticed that we’ve launched a new product in the Dedupe 2.0 space: Permabit Cloud Storage.

A few months ago, I made some snide comments about the “cloud” storage term. I’m not back-pedaling here, I still think that “cloud” is really “Storage as a Service 2.0″, but I do believe infrastructure is now mature enough that renting a home for your bits is now a viable thing to do.

The trap that I pointed out in my last message on cloud, and the reason I think EMC’s Atmos is misguided, is the problem of API proliferation. There are now dozens of simple, RESTful APIs out there competing for application adoption, and each locks you into a particular set of service providers. These standards will eventually coalesce and evolve, but in the meantime cloud providers are aggressively competing on the simplicity and power of their APIs… all while using pricey storage on the back end. Permabit Cloud Storage delivers massively scalable storage for these service providers that provides all the rich functionality they need to offer new web APIs, while staying agnostic on these interfaces by internally delivering conventional NFS and CIFS. Multiple service providers now use Permabit Cloud Storage to offer cloud services to their customers, freeing them from concerns over reliability, privacy and security.

Over at Storage Switzerland, Joseph Ortiz lays out the case for why cloud archive storage makes sense for small-to-medium business that have rapidly growing storage requirements. Permabit Cloud Storage provides infrastructure that allows service providers to deliver unbeatable pricing to these customers, as well as scalable systems that can be deployed as an internal corporate cloud. This is just one part of Permabit’s Dedupe 2.0 strategy — deduplication beyond the realm of just backup. There’s still more excitement to come.

The biggest dichotomy in deduplication is dedupe for backup, be it VTL or disk-to-disk (D2D), and dedupe for general purpose primary or archive data storage. These uses are very different. In the case of dedupe for backup the system doesn’t have to work as hard to find commonalities in data so it can “cheat” a bit when looking for identical data. This is largely because dedupe for backup depends on telling customers to keep on going with their same old inefficient backup schedules of regular full backups, even though those backups are now going to spinning disk. It’s really easy to get 20x dedupe when you tell the customer to store the exact same data 20 times!

At the heart of any deduplication system is an index of data that has been seen before, catalogued by some form of “data fingerprint”. (Permabit uses SHA-256, the only fingerprint algorithm currently allowed for Federal data security.) As new data comes in the system must rapidly fingerprint the data and determine if it has been seen before. Doing this quickly and efficiently is the core of any deduplication engine, and is an extremely hard problem to solve for hundreds of terabytes or more of data.

How does backup dedupe cheat? Dedupe systems for backup, like those from NEC, IBM, Sepaton and others, depend on something called temporal locality — a technical way of saying that because these are backup images, data that was written together before is likely to be written together again. A full backup of a system is going to look a whole lot like the last full backup of that same system, or a full backup of a very similar system.

Back to the deduplication index: Because these backup systems know that they’re going to see the same data over and over they don’t try to keep an index of all the data stored. Instead they break it up into smaller chunks based on time, covering only a handful of terabytes of data, or maybe a dozen backups. When they see a new backup stream start they look and see if it matches any of those small indexes — if it does, then they read in that index and use only it for deduplication. This catches a lot of duplicate data, but misses anything that was already seen in a different period. In practice, it works well enough for backup…. but, it only works well enough for backup.

Dedupe for Archive and Primary

For archive and primary data storage, on the other hand, the storage system needs to search hard to find duplicate data, looking for not just whole files that are the same but also smaller parts of files that are duplicated as different versions of that file. This is a much harder problem because to get any real efficiency the system must compare against all other data in the system in detail, hundreds of terabytes or more of data. Without inspecting all data in the system, significant deduplication will be unlikely.

As you might imagine, maintaining such a deduplication index of this scale is an extremely challenging prospect. Most vendors that claim to server archive and primary storage take the easy way out — they just don’t scale. Data Domain, for example, tops out at around 30 terabytes of disk, and NetApp won’t let you dedupe outside of a single 16 TB WAFL volume. You’re not likely to get much deduplication there! Others, like NEC’s HydraStor, will apply the backup-specific approach I describe above and happily charge you primary storage costs for archive storage with minimal if any deduplication. Bolt-on solutions like Ocarina are really only just recompressing your JPEG images more efficiently, and require custom reader software to go back and decompress them later.

Only Permabit’s Scalable Data Reduction (SDR) delivers truly scalable data deduplication for non-backup data. Our advanced grid architecture allows us to protect data far better than RAID by use of our RAIN-EC data protection, and also allows us to distribute the deduplication indexing problem. This means that Permabit Enterprise Archive can efficiently index and dedupe across hundreds of terabytes of disk — all real-time. In just thousandths of a second we identify if each incoming chunk has ever been seen before, something that no other vendor on the market today can claim within an order of magnitude.

Don’t get stuck with a deduplication solution that doesn’t perform. Before making your next storage purchase decision, ask your vendor how broad the scope of their deduplication is. Do they dedupe sub-file? Do they dedupe against all previously written data? What are the limits on the amount of unique data that they can store? The answers may surprise you.

I found it interesting, but not surprising, that they have better luck reading very old media versus newer failing media. While this might seem counterintuitive at first, I think it makes a lot of sense. Old data storage mechanisms were very simple to understand once they were invented and built. I remember but ten years ago when disk drives didn’t do any sort of automatic remapping of defective blocks — instead the drive came with a label identifying the blocks that tested bad, and you had to enter this into your computer so it knew to avoid them. Today, that’s encoded on the drive and much more sophisticated software would be required to reconstruct data from just the raw media.

Similarly, the magnetic domains on today’s storage are hundreds of thousands times smaller than 20 to 30 years ago. With today’s read technology we can recover media that has suffered significant damage, but there’s just so much less damage modern media can sustain and still be readable.

The Computer Museum’s problems are largely related to abandoned media that has been unmaintained and ignored, though, and Spicer doesn’t give hope to modern users who are still writing and preserving data. For users today the situation is not nearly so dire! Media migration and data preservation can be automated, as with the future-proof media migration built in to every Permabit Enterprise Archive system.

I last wrote about this in the first article in my data preservation series, No Silver Bullet: Archive Challenges. Organizations like the National Archives and Records Administration (NARA) recommend copying archive data to new, modern media every three to five years; this solves both the danger of media degration through media refresh, and the danger of media obsolescence through technology refresh. With large archive data sets this means that media refresh is occurring on a nearly continuous basis — there’s always some data on media reaching the end of its lifecycle.

Addressing this manually is a time-consuming, error-prone and painful process, as discussed in Pogue’s interview. That’s why we’ve automated it in our systems. New storage nodes can be added at any time with the latest and greatest technology, and older nodes can be removed as they reach the end of their life. All data movement is handled automatically, and data is never at risk during any of these internal migrations. This ensures that, regardless of how many petabytes of data you have, you’ll never run into the bit rot problem with a Permabit Enterprise Archive.

An entirely separate issue that Pogue doesn’t talk about in his interview with Spicer is the problem of logical readability. Just because you can get the raw data back doesn’t mean you can make sense of it. You may not have an application that can read it anymore! This would be like playing one of Edison’s robust wax cylinders only to find that it contained a message in a dead language! More on how to solve that in No Silver Bullet: Logical Readability.

Patents are written in a strange dialect of English colloquially known as “patentese”, so it’s hard to casually tell what they’re about — this is especially true of patents in high-tech. It’s easy to read a patent that’s for a better mousetrap (even if it’s titled “Mechanism for detecting presence, dispatching and retaining murine pests”), but what is “Storage system for randomly named blocks of data” or “History preservation in a computer storage system” about? Allow me to provide a bit of a secret decoder ring.

Records Retention

We have a bunch of new patents, so I’ll focus on two of the most interesting ones. The most recent is US Patent No. 7,478,096, “History preservation in a computer storage system”. If you click through you’ll see a lot of confusing wording, but the place to begin is at the start. There you’ll see “a method by which a disk-based distributed data storage system is organized for protecting historical records of stored data entities,” followed but a list of steps and components to make it clearer what we’re talking about.

From very early on, Permabit has included advanced features for records retention in Permabit Enterprise Archive. Things like WORM storage and retention policy management solve critical business problems such as complying with government regulations including SEC rule 17a-4 and FDA 21 CFR Part 11, both of which require data integrity and enforced retention of records. Until recently, such records were retained electronically only on write-once optical disk, or write-once tape. These solutions didn’t provide the accessibility required today to meet litigation discovery requests or to perform data mining operations.

Permabit Enterprise Archive was one of the very first products to allow enforced WORM retention of records on magnetic disk, and we filed key patents on the enabling technologies. This ‘096 patent protects our grid-based storage technologies that enforce records retention, be it through a file share or an object interface. Multiple versions of files can be stored over time, and even different versions can be protected and preserved.

Scalable Deduplication

The other patents I’ll discuss today protect some of the technologies we’ve developed that deliver Scalable Data Reduction, our efficient, in-line deduplication mechanism. The relevant patents here are US Patent No. 7,457,800, and it’s partner ‘813. The description here is a little less transparent, “storage system for randomly named blocks of data.” What’s this have to do with deduplication?

To understand, you need a very quick overview of how hash-based deduplication works. When data is being written into Permabit Enterprise Archive, our SDR technologies break that data up into smaller chunks for deduplication purposes. We then must very quickly identify if any of those chunks have been previously stored. We do this by taking a content fingerprint, or cryptographic hash — today that is the SHA-256 hash function, the integrity of which I discussed in a previous blog entry. Then we look and see if we already have a chunk with the same fingerprint.

It turns out that it’s really hard to do this when you have lots of data, and that’s the fundamental limit to scaling for archive deduplication systems. If you have a 100 terabytes of information you might have 10 billion different chunks, each with a 32-byte long name. That’s 320 gigabytes just of names! You can’t use an old-fashioned file system to store those and check, as it will take many seconds just to walk through the directory structure… that’s why companies like NetApp have to have background processes that just can’t keep up. Even a traditional database can’t do this quickly because all the names are evenly (randomly) distributed, so you can’t predictively cache part of the list. Think of it like a dictionary — just because you looked up “aardvark” doesn’t make it more likely you’ll look up another word beginning with “A” next.

To solve this problem companies like Data Domain have used computer science data structures called “Bloom filters”. These make it possible to scale to tens of terabytes, but break down as you go higher, so we developed our own technologies to scale out further, the “delta indexes” described in our patent.

This is one way in which we can scale dedupe beyond what other vendors can even dream of today. The other component is our decentralized grid architecture. I’ll explain how that helps SDR in a future post.

The Patent Pipeline

As I said up top, many of these patents were filed in the very early days of the company and are just finally making their way through the patent office now. Well fewer than half of our filings have completed the process to date, so I expect to have more good news to share in the future!

For this prediction, we say:

RAID Nears Retirement. As multi-tiered storage continues to evolve, SANs will become more complex, unified networks will emerge, and as newer and larger drive technologies such as 1 TB drives take root, RAID as a data protection technology will become irrelevant. Advanced data protection schemes based on Erasure Coding technology for long term reliable data storage will take hold putting additional pressure on legacy solutions depending on RAID.

RAID is a technology that has served us well, but there are two ways in which it fails to scale going forward. Most importantly, RAID technologies today have serious problems with large capacity drives, like the 1 and 1.5 TB drives shipping now. These problems will only become more pronounced with the 2 TB drives soon to be available.

First, RAID has an issue with the bit error rates on high-capacity drives, a problem I discuss in detail in our video “The Trouble with RAID”. The bit error rate is the rate at which a drive will fail to read a block. These failures are not due to complete spindle failures, but due to the statistical encodings used to store bits into magnetic domains on the drive. Drives don’t incur the penalty of read-after-write to verify the data written, so sometimes they manage to write data that cannot be read later despite the sophisticated error correcting codes used to protect the data on disk.

As I explain in an earlier post, the bit error rate of the drives can be catastrophic for RAID. In a RAID 4 or 5 rebuild it is necessary to read every bit off all the remaining disks. There’s a high probability this may not be possible with high-capacity drives. In RAID 6, the same problem occurs in the event of a double failure.

This very problem was raised at the recent Gartner Data Center Conference, in “The Enterprise Storage Scenario” by Roger Cox and Dave Russell. RAID is not a technology that is going to survive with higher and higher capacity drives, and enterprises must look to technologies like advanced erasure coding to meet data protection requirements.

Permabit Enterprise Archive protects against this pitfall within our RAIN-EC storage architecture. By recording additional recovery information we can rebuild from up to 8K of unreadable data without having to fail a drive or recover from another location. Even in the event of multiple failures you’re still protected against the bit error rate, something that RAID can’t do.

The second problem RAID faces are increased rebuild times. While drive capacities continue to grow exponentially, drive read performance does not. The read rate is dependent upon drive spindle speed and linear bit density. Large capacity drive spindles aren’t spinning any faster, with all of them in the 5400 or 7200 RPM class. Bit density is going up, but read rates only improve with the square-root of the rate at which capacity increases, because capacity gains from increases in two dimensions (around the disk and across it) while read rate gains from only one.

This means that RAID rebuilds take unacceptably long times on high capacity drives. Consider a rebuild at 25 MB/s for a set of 2 TB drives — this will take more than 22 hours! Can your data be without protection for nearly a full day?

Permabit Enterprise Archive’s RAIN-EC architecture helps here as well. While in a RAID system a group of drives constitute a set, RAIN-EC distributes data in a more sophisticated manner. The recovery information for data on one drive is spread evenly across all the other drives in the system. This means that in the event of a drive failure all the other drives participate in the reconstruction process, and each drive is only responsible for a small portion of the recovery. Thus, the rebuild rate goes up with each additional drive in a RAIN-EC system. With RAID, adding more drives always makes the rebuild rate go down (or stay the same).

That’s the pressure from the high capacity side, but RAID arrays, at least for disk, have serious pressure from the other side too — Solid State Drives (SSD). SSDs massively outperform low capacity 10K and 15K RPM drives, and within 18 months they’ll be at an equivalent price. Additionally, STEC tells me that their Zeus SSDs have bit error rates as low as 1 in 10^17, which protects in rebuilds significantly better than the equivalent 15K RPM drives.

Given reliability concerns when using high capacity disk drives and the end of road in view for 15K RPM performance-oriented disk, RAID arrays are being squeezed from both sides. High performance systems will continue to use similar technology on SSD, but archive systems require more advanced technologies for the future, and the future, as always, is sooner than you think.