The Ballad of AWS Storage Gateway - TriNimbus

The Ballad of AWS Storage Gateway

A few months ago I had an interesting chat with an old buddy of mine over a pint of lager at our neighbourhood pub. We of course started our conversation around families, kids, vacation plans…but, since we both shared a background, our conversation predictably turned to its usual path.

I was enthusiastically talking about the new services and features AWS revealed at the 2016 re:Invent conference when I realized that my friend was looking at me with this strange, rather tired, look like something was bothering him. I took my cue and asked if there was a problem.

He explained that the board of the company where he worked decided to start cutting costs across the entire organization based on the organization’s financial performance. All the department heads were tasked to come up with ideas for cost saving initiatives, including headcount cuts and slashing of departmental annual budgets.

Having had conversations with him for the last few years, I didn’t need to be a management guru to understand that potential reduction of even one resource in the department would very quickly escalate to an enormous pressure and impossible workload on the rest of the team.

To make matters worse much of the company’s IT infrastructure, such as production storage arrays and backup systems were approaching 8 years in service, and needed a budget allocation for a refresh before failures added more stress to the situation. He was basically locked in a catch-22 situation.

This scenario is nothing new, and every company that is not in the cloud faces the same issue once every few years when the time comes to phase out aging infrastructure. The challenge is even greater though when the business sees IT as a cost, rather than an critical investment essential to the business.

Being familiar with the field, I already knew that there are many different possible solutions available to help him address his company’s issues with hardware. Having done a few projects with storage migrations to the cloud, I knew the scale, costs, time and the process for adopting a cloud solution. Being familiar with my friend’s pain points, the right solution for his storage and backup woes seemed quite obvious to me, and that is when I decided to introduce him to AWS Storage Gateway.

AWS Storage Gateway is a service that seamlessly and securely integrates on-prem environments with cloud storage and consists of a few components:

  • Virtual appliance that is being deployed to your local VMware or HyperV cluster

This appliance can be deployed emulating multiple entities, such a local file server leveraging NFS or iSCSI protocols, or a Virtual Tape Library that can be seamlessly used in conjunction with your Backup Application.

  • AWS Managed Back-End – that is responsible for the management of the service, establishing SSL tunnel between the backend and on-prem appliance and transfer of the data to the storage back end.
  • Highly scalable and durable cloud storage back end (S3, Glacier, EBS) where the data is being stored, archived and backed up

Storage gateway comes in a few flavours:

  1. File Gateway is basically a file interface into Amazon S3, where files can be stored and retrieved directly using NFS protocol. The same files can be accessed directly in S3 from any cloud application or service, and also the same data can be managed directly in Amazon S3 using standard S3 tools such as lifecycle policies, cross-region replication, and versioning.
  2. Volume Gateway – provides cloud-backed storage volumes that you can mount as iSCSI devices from on-prem application servers. The volume gateway can be deployed in two configurations: cached volumes and stored volumes.
    1. Cached volumes – In this scenario the data is being stored in Amazon S3 and a copy of frequently accessed data subsets is being retained locally. This approach offers a substantial cost savings on primary storage and minimizes the need to scale storage on-prem. There is also low-latency access leveraging the local cache to the frequently accessed data.
    2. Stored volumes – This method provides low-latency access to the entire data set. The on-prem gateway is configured to store all the data locally and then asynchronously back up point-in-time snapshots of this data to Amazon S3. This configuration provides durable and inexpensive off-site backups that can be recovered to the local data center or Amazon EC2.
      This method could be a perfect fit for a solution that requires local storage with low latency local connection while async mirroring the data to AWS Cloud for DR purposes.
  3. Tape Gateway – Using VTL configuration of Storage gateway, the data can be backed up to S3 and later durably archived in Amazon Glacier. Tape Gateway provides a virtual tape infrastructure that includes virtual tape appliance, virtual tapes, media changer, tape library with up to 1500 slots, and 10 tape drives, that scales seamlessly with the business needs and eliminates the operational burden of provisioning, scaling, and maintaining a physical tape infrastructure.

The Total Cost of Ownership of this solution varies and depends on the architectural design, amount of data and storage tier chosen for the deployment, but overall AWS Storage Gateway is extremely cost effective when compared to any other enterprise grade storage solution, and can be provisioned and deployed within minutes instead of weeks or months in comparison to the traditional storage systems.

To make the long story short, after a couple of months doing a proof of concept and testing AWS Storage Gateway, my friend’s company deployed it to to their production environment and replaced aging storage arrays with zero upfront investment, with a very short learning curve for his IT folks and the annual OpEx associated with that solution was just a fraction of costs of the traditional Enterprise-grade storage, backup and archival solutions, that otherwise would have had to be acquired.

If you read this and find yourself facing a similar hardware refresh on the horizon, take the time to carefully consider how the Cloud fits in your long term plans. Strategic plans to migrate hardware solutions to the Cloud can be daunting to start. If you need help with planning and assessing your Cloud strategy, TriNimbus would be happy to assist.