Spring Cleaning Your AWS Account - Ideas and steps you can take to reduce the clutter and cost - TriNimbus

Spring Cleaning Your AWS Account – Ideas and steps you can take to reduce the clutter and cost

I spent last weekend doing some spring cleaning. Granted it’s not the most exciting weekend activity, but once a year isn’t too much time to spend getting at those far back corners collecting dust that I never think about. I also got rid of some extra junk that was just taking up space. By the end of the weekend I had less clutter, could breath easier,  and felt some accomplishment knowing that I was in a better place (for a while anyway).

While your AWS Account has less dust bunnies than behind my desk, the same principals apply. Every now and then you should give it a good spring cleaning to declutter, organize and hopefully save a little money.

Most of the year I look at our AWS Invoices to see what has changed and if there are any significant cost increases because something was left running. However I usually focus on the larger cost items only, despite the fact that ongoing costs for smaller items can add up over time.

For example, let’s consider a scenario where we implemented a proof of concept test for an application that had 3 public application servers, 2 private application servers and a MySQL database. A static IP address was assigned to each of the public servers, an internal load balancer was used for the private application servers, and a VPN connection was setup to safely load data into MySQL from another data center.

When the proof of concept was completed, all the instances were terminated after a final snapshot was taken in case further testing was needed.

Unfortunately, the project never progressed and the final cleanup was forgotten about. This means we have the following resources not being used:

  • The final EC2 instance snapshots were left, even though they are not needed.
  • We left a Relational Database Service (RDS) instance, 3  Elastic IPs (EIPs), and an Elastic Load Balancer (ELB) running.
  • An idle VPN connection for loading MySQL data was left running.

In an environment running multiple production and non-production workloads, the above scenario would not be easily identified.  However these costs can add over the course of a year.

Unused Proof Of Concept Resource Costs

  • Allocated, but unassigned EIPs are billed hourly.

24 hours * $0.005/hr * 365 days = $43.8/yr * 3 EIPs = $131.4/yr

  • A t2.medium MySQL RDS instance using single AZ with 500 GB SSD storage.

Storage: 500 GB * $0.115 GB/mo * 12 months = $690.

On-demand Hours: 24 hours * $0.068/hr * 365 days = $595.68.

RDS Total: $1285.68/yr

  • An idle ELB:  24 hours * $0.025/hr * 365 days = $219/yr
  • EC2 instance EBS snapshots: 50 GB * 5 instances * 12 months * $0.095 GB/mo = $285/yr

Base cost per year for incorrectly terminated POC: $1921.08 USD

Note: This does not include taxes, exchange rates, and costs for AWS Support.

Without doing a periodic clean up, that $2,000 can keep being added to your bill year after year. You also have the extra data to filter out when you are trying to find specific resources.

With this in mind, there are two problems. How do you find and cleanup the mess and how can you prevent having abandoned resources hanging around?

There is a bit of an art doing a cleanup, much like cleaning windows without leaving streaks. You want to make sure you look at what you have in it’s entirety before you remove resources. If you were to just delete an old EC2 instance, you may leave behind EBS Volumes, Snapshots, EIPs, Route53 entries, or IAM Instance Profiles/Roles.

The AWS Config Service can provide a snapshot of some resources. You can also use the AWS CLI to get filtered results using the filters and query parameters to limit data. For example:

aws ec2 describe-snapshots  –filters  “Name=owner-id,Values=123456789012” –query “Snapshots[?starts_with(StartTime,’2016‘)== `false`].{ID:SnapshotId,Time:StartTime,Desc:Description,Size:VolumeSize, State:State}”  –output table –region us-west-2

 

The above command provides a list of EBS snapshots for an account from before 2016. It needs to be run against each region.

Once you have a list of resources to clean up, you can begin removing them. Here are some tips for managing resource cleanups:

  • First, look for high level resource collections to remove. For example CloudFormation Stacks, OpsWorks Stacks, Auto Scaling Groups, EMR Clusters and Elastic Beanstalk Applications. This will help remove all related resources.
  • When deleting EC2 instances check for attached EIPs/ENIs, assigned IAM Instance  Profiles no longer needed, and review block device mappings to see if non-root EBS volumes will automatically delete on termination. It’s also a good time to confirm the Termination Protection status of the instance to ensure it can be deleted.
  • When deleting instances and volumes that are no longer needed, remove any AMIs or snapshots first.
  • Instances may be using additional services like Route53 DNS, Elastic Load Balancers, CloudFront, and CloudWatch. Review to see if you can remove related resources.
  • Software systems may require updates when removing instances. Check if instances are connected to any directory service, inventory/license tracking system, or 3rd party monitoring systems.
  • The Key Management Service now charges for disabled keys. Before scheduling key deletions ensure no one else is using the key.  Review the key grants, IAM permissions and CloudTrail logs for insight into key usage.
  • Review Kinesis Streams and DynamoDB (DDB) tables for over provisioned capacity.
  • Review your usage of reserved capacity resources. For example, DDB Reserved Capacity, EC2/RedShift/RDS Reserved Instances and ElasticCache Reserved Cache Nodes. Look for missing renewals, opportunities to reallocate unnecessary resources and ensure that instance types and availability zones match the expected reservations.
  • Look at your scheduled CloudWatch Events for Lambda functions that no longer need to have scheduled executions.
  • If you have AWS Premium Support enabled on your account, use the Trusted Advisor service to identify underutilized resources.  It’s also a good time to review your service limits while you’re there.
  • Check your AWS Invoice costs in non-primary regions and investigate resources to ensure they are still actively required.
  • Take a look at your IAM Users to ensure all users are active. You can download a Credential Report to see who is actively using their account, and to ensure that credentials are getting properly rotated based on your requirements.

In order to prevent some of the clutter that your account can accumulate, here are a few suggestions:

  • Utilize Resource Tags to identify owners, project allocations, environment types, longevity status, and expected expiration date for resources where possible. AWS Config Rules can help with this.
  • Try to use services like CloudFormation to manage related resources as a collection.
  • Watch for long running on-demand resources. Resources running for several months might benefit from a reservation or may have just been forgotten about.
  • Consider implementing Object LifeCycles on S3 buckets to archive/delete old content.
  • Setup AWS Billing Notifications or use 3rd party tools like CloudCheckr to monitor and notify on cost trend changes.
  • Setup daily automation to track or notify on things like unassigned EIPs, EBS Volumes in an Available status, resources that have passed the date from expiration tags, EC2 instances with failed status checks, ELBs with no healthy instances, or snapshots older than 6 months.

While your spring cleaning and preventative maintenance on your AWS Account may take more collaboration and time focus than I had with my dust cloth, it’s still probably worth the effort. If you need help with prevention steps or cost optimization advice, TriNimbus can help get you to that better place.