Effective use of data reduction in the Public Cloud
Permabit CEO, Tom Cook recently wrote about how data reduction technology can simplify the problems associated with provisioning adequate storage resources in the public cloud, while balancing performance and efficiency. The good news is, taking advantage of data reduction software in the public cloud is easier than ever.
For example,Permabit’s Virtual Disk Optimizer (VDO) is a pre-packaged software solution that installs and deploys in minutes on Red Hat Enterprise Linux and Ubuntu LTS Linux distributions. To deploy VDO in Amazon AWS, you provision Elastic Block Storage (EBS) volumes, install the VDO package in your VMs and apply VDO to the block devices represented for your EBS volumes. Since VDO is implemented in the Linux device mapper, it is transparent to your applications installed above it.
As data is written out to block storage volumes, VDO applies three reduction techniques:
- Zero-block elimination uses pattern matching techniques to eliminate 4 KB zero blocks
- Inline Deduplication eliminates 4 KB duplicate blocks
- HIOPS Compression™ compresses remaining blocks
This approach results in remarkable 6:1 data reduction rates across a wide range of data sets. At Permabit we take advantage of these savings in our own AWS dev and test environment and, as Tom’s post notes, a 300 TB configuration, assuming 50% utilization time, can save $150,000 over a one year period. But that’s just the start!
The way data reduction is being applied at the block level means that the process doesn’t just apply to the active data, but also to snapshots. Data reduction of snapshots minimizes the cost of long-term storage in the public cloud adding to the estimate mentioned above. What’s more, the reduced data can be uploaded to other public cloud providers or downloaded to your data center (provided VDO is installed there) to allow access to the optimized format. In addition, this capability enables you to realize substantial bandwidth reduction when migrating data from one provider to another. The massive savings that result from these use cases can substantially boost the agility of your public cloud deployment.