The most important thing we learned in performing this study is that capacity savings are extremely dependent on how RMAN is used. To maximize deduplication efficiency when using RMAN, there are three things to keep in mind:

1. FILESPERSET=1

This setting suppresses Oracle multiplexing of more than one data file into a backup set. When you suppress Oracle multiplexing, Oracle creates the backup set identically each time the backup runs, maximizing the dedupe rate.

2. Compression/Encryption

Both compression and encryption must be disabled in RMAN to see effective deduplication. However, compression should be left enabled when backing up transaction logs and control files since they don’t see good deduplication rates anyway (see below).

3. Archive Logs and Control Files

By default, RMAN interleaves archive (transaction) logs and control files in with database content. This defeats deduplication because it changes the alignment of data. When used with deduplication, these additional files should be written to separate streams.

With these settings in place, our team performed two tests of dedupe efficiency with RMAN. 

First, we performed incremental backups. We created an Oracle database and populated it with 1 GB of randomly-generated data. This was followed by 6 incremental backups (simulating a full week of backups), each after an additional 100 MB of data was added to the database. Finally, a second full backup was performed. The results were scanned using the albscan utility, the dedupe assessment tool we include with our Albireo SDK. For this example, we saw up to 91% capacity savings in RMAN environments. 

As a second test, we again populated an Oracle databse with 1 GB of randomly-generated data. This time, seven full backups of the database were performed (simulating seven weeks of full backups), again each with an additional 100 MB of data.  For this second case, we saw a 94% capacity savings.

To put it another way, Permabit Labs saw a 10x increase in storage efficiency in just the first week of RMAN backups with deduplication. With retention of 7 full backups we saw a 16x increase. That’s some substantial short-term ROI!

With a long term data protection strategy it’s easy to project savings as high as 35x. For more information on data protection and Permabit’s Albireo High Performance Data Optimization products check out http://permabit.com/oem/products-with-albireo/data-protection/

For this discussion, we will focus on Oracle database environments (but keep in mind that this applies to other Database environments as well). Oracle backups are performed using the RMAN utility. When backing up Oracle databases, DBAs use a rotating schedule of full backups, which backup the entire database, and incremental backups, which backup only those changes made since the previous operation. Incremental  backups are more efficient to store and transmit, but DBAs cannot use them forever because they complicate the restoration process. Each incremental backup since the last full must be replayed back into the database increasing the time it takes to restore. For each full backup performed, an entire copy of the database gets written to a backup device , typically consuming as much (or more) additional disk space than the previous full backup. Backups data is typically groomed over time with older copies being deleted on a periodic basis depending on company policy. Many of those company policies are driven a combination of factors that include both data retention objectives and storage costs.

If that same data is sent offsite, over the network (for disaster preparedness), it consumes not only more disk space at the remote site, but also significant bandwidth. In this case, if that full backup is large, the time it takes to replicate it offsite can extend beyond a company’s backup window, increasing exposure to data loss in the event of catastrophic failure.

Permabit labs just recently completed a 6 week study on the effectiveness of the data deduplication technology provided by our Albireo data optimization product offerings when used with Oracle RMAN backups. In these tests, we were able to demonstrate data reduction rates of between 83% and 94%. This study proves that by taking advantage of Albireo technology, our OEM partners can help their customers to save significant costs on storage while meeting their disaster recovery objectives.

In follow on posts, I’ll discuss the methodologies that were used in performing these tests as well as the implications of applying Albireo deduplication technology in the database environment.

“Don’t they already have dedupe,” asked Tom Cook?

“Well yes,” replied the analyst, “but every one of them, even the largest and most established players, has issues with performance, resource utilization and/or scalability when compared to the leading data protection appliance companies. From what you’ve said about your technology, I can see a fit in any of these product offerings. You’d improve any product out there in at least one key metric by an order of magnitude.”

An order of magnitude? That’s interesting, I thought. This analyst talks with IT organizations AND vendors on a daily basis and we’ve known each other for some time. I’ve come to trust his advice over the years. But it’s frankly difficult to check out the performance, resource utilization and scalability limitations of enterprise backup vendors without kicking off a massive research project. If you’re in that space, or even if you’re in another area where you think you’ve got dedupe covered, maybe you should be asking these types of questions?

1. How fast is my solution with/without dedupe?  If your ingest rates drop substantially due to dedupe – Permabit Albireo can help!
2. How memory intensive is dedupe in your solution today?  If you’re consuming more than 256 MB of memory to dedupe 10 TBs of unique data – Permabit Albireo can help!
3. How scalable is my solution – can I dedupe across multi-system pools of storage or is my solution limited to addressing specific media silos? Can I scale beyond a single server? 2 servers? 16 servers? 128?  If you can’t utilize commodity servers to scale to PBs of storage – Permabit Albireo can help!

To learn more about the capabilities of Permabit Albireo versus other deduplication technologies, check out this series of articles by Wayne Salpietro, Permabit Director of PMM and be sure to visit our newly redesigned website at www.permabit.com.

Much harder to predict are the benefits of space savings seen within different data workflows. This is where deduplication can offer huge benefits! For example, an email attachment might be sent to an individual or it might be sent to a group of people. In the first case, you won’t see much savings, but in the second the savings from deduplication are substantial.

Early on we learned a simple truth – understand how data flows and you will better understand the benefits of data optimization technology in a given environment. In fact, we found a number of common data workflows that benefit from data optimization technologies and particularly from data deduplication. The flow of email attachments I mentioned above is just one example. I discuss two others related to data workflows associated with VMware deployment and enterprise database management in previous posts.

But let’s take this a step further. What if the greatest of these opportunities for savings come from data workflows that don’t even exist today? Data management workflows that we wouldn’t even consider without dedupe? What is the value of the fundamental capability, being able to keep extra copies of a dataset without having to pay for corresponding disk space? How will it impact the workflows of IT organizations and their end-users? As capacity optimization technologies have emerged, IT organizations have taken them in many different directions, substantially expanding their value. Take storage snapshots as an example. The original purpose of this now ubiquitous capability was to maximize application availability during backups. However, it didn’t take long for IT to take the technology in a different direction. Soon IT organizations were utilizing snapshots to provide user-initiated data recovery services. The end result freed up already constrained IT resources.

Like snapshots, data deduplication emerged from the data protection trenches to be deployed in primary storage environments. One can only wonder what common-sense practices become feasible as more and more optimized storage solutions are deployed. Will we see a marked decline in end-user requests for disk quota increases? Probably not, but perhaps we will see software developers adapt applications themselves to take advantage of deduplication, exposing more configuration management, versioning and data recovery capabilities directly to end-users. Do you have any ideas of how primary dedupe might impact the data workflow of your organization? Send me your comments and I’ll follow up in a future post.

We know there isn’t exactly a huge field of players in primary storage dedupe. You can count them all on one hand at this point! One place where primary storage deduplication gets tricky is in regards to the amount of resources required for effective performance. This gets into my second point from last week.  For primary deduplication to work for enterprise databases it has to be cost effective. At least one shipping filesystem technology out there requires between 16GB and 32GB of RAM just to perform at primary speeds and deduplicate a single TB of raw data using 4K blocks! For anything but the most fantastically expensive electronic memory-based storage, this is not going to be cost effective. Even with a 20:1 dedupe rate, who would use it?  With the cost of 1TB of enterprise disk space today matching the cost of 1GB of DDR3 memory where’s the economic advantage? A vendor might try to get back some of the performance they’ve lost by increasing disk speeds by storing their index on an SSD. But SSD access is much slower than memory, and SSDs are expensive too!

In comparison, Albireo uses a patented index technology that allows it to offer full performance in a fraction of the memory footprint of competing solutions. Today Albireo is able to run with ¼ of the resources of the most advanced competing technologies out there, and that efficiency is only going to increase over time as we release new versions. Given Permabit’s enormous lead, I don’t see other companies catching up anytime soon.

Scalability is also a big inhibitor for the majority of the folks I heard from this week. It seems everyone who is using primary storage deduplication with enterprise databases today is doing so with relatively small databases. Why? It’s due to the capacity limits of the deduplication feature. If you’re storing copies of an inventory control database, a dedupe solution that supports a couple of TBs of deduplicated data can be plenty effective. With the folks I heard from, capacity limitation was the most common complaint – they liked the savings, but couldn’t use the solution for all of their DBs. Some of their databases were simply too big! As Albireo-based solutions start to hit the market, you’re going to see these limits increase to hundreds of TBs. They are where they are today because of resource constraints in the hardware similar to the ones I mentioned above. As I noted already, the Albireo technology allows storage vendors to leapfrog their competition by providing substantially larger deduplication capacities at a fraction of the cost.

So there we have it. Today performance, scalability and cost are the main inhibitors to the adoption of primary deduplication technology in the enterprise database environment.  These issues will be addressed by the new Albireo enabled storage products as they emerge. Reduced storage costs and improved development efficiency are the inevitable outcome.

I was surprised to hear a DBA tell me how he’s seeing tremendous benefit (10X savings) from a primary dedupe solution today. What I missed in the cost/benefit equation were all the additional copies of the data that get utilized for development, standby and test, let alone the copies eventually used for historical purposes and/or offloading for more efficient query functions. Of course! Database copies are all over the data center, not just in backups. In fact, the modern grid-based infrastructures delivered with today’s database solutions actively encourage their proliferation by greatly simplifying deployment. So instead of one copy of each database you have 7 to10, near identical copies, each of which can take advantage of deduplication and provide substantial space and cost savings.

I see three reasons for lack of adoption of primary storage deduplication with enterprise databases today and they are the classics: performance, cost and scalability. I’m going to start by talking about performance this week and post additional blogs on the other topics over the next few weeks.

My background is in operating systems development, so I’m well aware that the typical enterprise database works with fixed blocks of data and relies on temporal alignment of those blocks on fast (tier 1) disk to help reduce latency and ensure performance meets SLAs. It makes perfect sense that this type of architecture will see almost no incremental overhead from deduplication as enabled by Permabit’s Albireo because the approach typically leverages the extent management processes that are highly efficient.

Some other primary deduplication solutions out there today are sitting in the data read path. This has the inevitable effect of introducing additional latency on write and also on read. So these vendors tell their customers upfront, ‘not for structured content’ and DB is not even considered as a use case – probably a good thing too! As I’ve already pointed out in my recent blog on VMware Performance and Deduplication, Albireo’s high performance data optimization sits entirely outside of the read path and is designed to keep up with the ingestion demands of modern block and filesystem based storage solutions. Sitting outside the read path is key for the database administrator and basically means if a storage tier is suitable for an enterprise database deployment, it’s suitable with albireo deduplication.

I’ll write more about the cost and scalability issues next week.

Albireo’s unique design provides efficient deduplication without any performance penalty because Albireo runs entirely out of the read path.  Albireo’s role is to provide deduplication advice to existing storage subsystems. Simply and elegantly, Albireo takes advantage of the built-in capabilities of modern file system and block virtualization layers provided by all mainstream storage vendors.  With this design, Albireo creates no more performance impact than when common features such as clones and snapshots are in use on the storage system.

So what about ingestion speeds?   This is where Albireo shines.   The grid-based capabilities of the Albireo index allow it to be scaled across multiple independent CPU resources to provide deduplication advice on ingest at breakneck speeds.  While Albireo is rated in excess of 140 MB/sec per processor core on a 3.0 GHz Intel Xeon even when operating with a small 4 KB chunk size, performance is even greater with larger chunk sizes. With 64KB chunks we’ve seen Albireo Grid-based dedupe scale to support ingest speeds over 5.5GB/s.   These speeds are more than adequate to meet the needs of even the most intensive VMware environments. So there is no performance impact or latency as has been identified in early stage deduplication backup offerings.

What’s more, Albireo allows storage vendors to give their customers choice.  With Albireo, an OEM can chose to implement  deduplication inline, in parallel, and/or as a post-process event – providing the greatest flexibility for performance intensive environments while simplifying manageability.  This means that in the latest powerful storage equipment deduplication can be performed inline with no performance impact, while in more resource constrained storage hardware, customers simply schedule deduplication to run at off hours.  Either way, there is no performance impact to the user environment.

Finally, I wanted to talk a little about a way in which deduplication can actually improve read performance.   While Albireo itself doesn’t sit in the read path, it does provide advice which allows blocks in the read path to be shared.   Like VMware Transparent Page Sharing (TPS), Albireo looks at new data and provides advice on how to eliminate duplicate blocks using existing capabilities of modern storage systems. These existing capabilities maintain deduplicated blocks in the read cache, reducing the amount of memory needed for caching the operating systems and applications stored in VMware environments.  The result is improved overall performance when reading these shared blocks from disk.   This becomes especially important in virtual desktop deployments, where many users logging in at the same time can result in boot storms.  In these environments, deduplication can reduce the impact of boot storms by up to 50%.

Deduplication systems work by identifying and eliminating duplicate chunks of data. With Albireo, deduplication gets implemented by the OEM at the shared-storage, logical volume or filesystem level – the larger the shared pool of data on the back-end, the greater the potential for savings from redundant data. Albireo’s flexible deployment options enable deduplication to occur inline, parallel or as a post-process event to address varying performance requirements.  We’re working with some OEMs who are implementing all three options on the same systems.  Regardless of where or when deduplication is implemented, the process is the same; duplicate chunks are eliminated and data pointers are modified to share a single data chunk on disk.   Albireo’s job is to identify those duplicates as quickly as possible, while utilizing as few system resources as possible!

To help simplify the management of consolidated environments, VMware server virtualization allows administrators to create ‘template operating environments’, each with a standardized operating system and application environment. These templates are then ‘cloned’ into separate VMware images and installed as ‘guests’ on a physical server. The result is savings from simplified configuration management. Today, with modern processors from AMD and Intel, I’m seeing VMware users running modern server environments with 12 VM guests per physical server. I have heard of desktop virtualization environments with over 60 ‘guests’ per server. While VMware substantially reduced server costs, it has done nothing to help storage administrators who are left with the same number of operating system images on what is now a complex mixture of shared-storage resources.

It’s important to realize that while VMware provides a valuable cost benefit from consolidating IT servers, it offers no means for consolidating the storage used by VMware ‘clones‘.  That’s where deduplication comes in. You see it turns out that each of these ‘cloned’ VMware images starts off by taking up as much space as the template from which it was created. The set of data chunks making up a ‘template‘ and each of its ‘clones’ are nearly identical. When Albireo see’s these duplicate ‘chunks’ it identifies them and allows the storage system to replace them with pointers to the original data. Its easy to see why in a 12 VM guest environment, there’s nearly a 12:1 savings even after accounting for personalization of each guest. For a 60 VM Desktop environment, there’s a potential for almost 60:1 savings!

Next week I’ll talk a little about how Albireo can actually help VMware administrators better meet their storage performance requirements.

1. We were able to reduce the overall cost/GB of our equipment by 20% from previous models
2. We cut the power footprint by 28%
3. Our relationship with HP allows us to offer HP onsite service

Let’s take a look at the cost savings.

Permabit storage is grid based, utilizing a set of independent server nodes to provide redundancy. As new generations of these nodes become available, customers can add them into a grid with their existing Permabit hardware and leverage the latest improvements in server efficiency. By basing our new nodes on the HP Proliant SL6000 series, we’re able to have two independent hot swappable servers share a common chassis, redundant power supplies and fans. This saves costs and provides redundancy. We are also able to support up to 6 drives per U. With six 1 TB drives and a typical 3X deduplication rate, that’s like providing 18 TB’s per U of raw storage! All the component sharing and increased density translate to 20% savings in equipment costs/GB, even before savings from dedupe, not to mention the savings in shipping costs (these new units weigh 30% less).

So the cost benefits are tremendous, what about power savings? As you might suspect, shared power supplies and fans can yield significant savings here. But over and above that, these units have front IO cabling, not just to improve serviceability, but also to increase airflow. Since the only cables in the rear of the modules are the power cables, the back is open for heat to exhaust and not get blocked by cables. Because of this unrestricted airflow, the fans are more efficient at cooling. The overall result is a 28% reduction in power and cooling costs.

On the service front, Permabit continues to provide 24×7 remote support services to our customers along with our “phone-home” monitoring capabilities. What is different now is that onsite hardware service is provided by HP. Standard support for the new HP units from Permabit includes 3 year Next Business Day onsite support from HP with upgrades available to 24×7 onsite support options. As the worldwide leader in server shipments, HP is typically already in enterprise data centers so maintenance may be done by the same HP personnel you are already working with. If you’re not already an HP customer, that’s ok too! Permabit Support handles the dispatch process and completely manages your call from end-to-end.

The customer benefits of the addition of HP to our technology stack are significant and we are excited to provide better cost effectiveness, serviceability and a brand name that provides excellent support.

Coming in with our background, secure protection was seen as a key requirement for any enterprise archive storage solution. Permabit went into its “enterprise archive” project with 5 years of experience in developing compliance and governance technologies for enforced information retention on disk-based storage. Both Write Once Read Many (WORM) and Encryption are used to protect an archive from malicious or even accidental user interactions. These capabilities are critical for data that is being retained for regulatory compliance and they equally, if not more, are important for legal risk mitigation. Under current regulatory guidelines, organizations must be able to prove that any information relating to a case does not change (chain of custody). The easiest way to prove chain of custody is with WORM technology. We built into Permabit Enterprise Archive the most flexible WORM functionality available with read/write convert to WORM, Compliance WORM to address stringent immutability requirements and Enterprise WORM with the flexibility to delete WORM files.

In my next blog, I’ll talk about how Permabit launched the Enterprise Archive product and provide a brief overview of benefits seen by Permabit customers today.