This article will specifically highlight various strategies for optimizing costs in AWS.
Complex multi-cloud environments are becoming commonplace, and billing details vary dramatically by provider. It’s not uncommon for Amazon Web Services (AWS) customers to receive higher-than-anticipated bills for cloud services. While this might be expected when organizations first migrate workloads to AWS, it can be frustrating for organizations that have rightsized all of their instances and taken advantage of every discount opportunity available. Sometimes, unexpected charges can be attributable to the enthusiasm of development or CloudOps teams who, unaware of the budgetary implications of their actions, may want to help drive the organization forward by experimenting with new services they’re not yet familiar with. This kind of behavior requires further investigation and additional efforts to prevent it from becoming an ongoing issue.
However, in many circumstances, unexpected charges on your AWS bill can be attributable to a lack of visibility into cloud resources and activity, which can result in services being used without approval, costs accrued without warning, and charges related to resources that are no longer in use or required.
1. Shadow IT Activity – Shadow IT occurs when users, teams, or departments deploy resources or embrace new processes without proper approval and sufficient understanding. AWS Cost Explorer has a wide variety of filters which can help you identify rogue features and services and Amazon Cloudtrail tracks API usage across your entire infrastructure, which can help you to track down when, where and by whom each service is used. Service control policies and IAM roles allow you to whitelist services, preventing trigger happy developers jumping on the latest and greatest services and driving up huge costs. Until you have total visibility across your environment and have implemented measures to prevent the use of unsanctioned services, shadow IT costs are likely contributing to your AWS bill.
2. Greedy Configuration – Many AWS services have been designed with ease of use in mind. AWS allows its’ users to quickly build, connect and deploy sophisticated solutions, by combining a wide variety of services that ‘just work’ out of the box. However, it is important to understand the underlying configuration and the impact it can have on your bill. How much memory does your lambda function have? And how much does it need? Millions of invokes of a lambda with 10GB of RAM is going to cost more than the same lambda with 128mb or 256mb of RAM. SImilarly, how much disk space, if any, does your ec2 instance require? Attached volumes and regular backups can quickly increase your bill. Look at things like backup and encryption options for any service you use, are either or both required?
3. Servers – When moving from an on-prem solution to the cloud equivalent, the path of least resistance is often replicating the solution. Using EC2 instances, EBS volumes and provisioned databases are a great first step, but consider whether migrating to on demand serverless services like Lambda, Aurora Serverless, DynamoDB and S3 are a viable option. If you decide you do require servers, look at Ec2 spot instances and AWS Fargate as potentially viable cheaper options.
4. Inefficient Scaling – Vertical scaling involves taking a single resource and allocating more resources to it, although it can solve some resource consumption issues in the short term, it quickly becomes very expensive and eventually becomes impractical in terms of scale. AWS Auto Scaling allows you to scale horizontally by distributing your work load across multiple on demand instances. In theory it enables you to maintain steady, predictable performance at the lowest possible cost. In reality, it’s rarely that simple. It’s common for teams to take shortcuts when configuring auto-scaling groups that lead to larger capacity-to-demand ratios, and consequently, lead to unnecessarily higher costs.
Don’t make the mistake of thinking auto-scaling groups will do all the work for you – it’s important to have access to granular metrics about the initial configuration and ongoing utilization of auto-scaling groups in order to accurately identify and optimize inefficient or wasteful configurations. Use AWS Cloudwatch Events to trigger scaling events, AWS Cloudwatch Insights to monitor your scaling activity and use AWS Cloudwatch Alarms to alert any abnormal or unexpected behaviour.
5. 24/7 dev environments – a common architecture approach to cloud development teams is to replicate a production environment in a development account, once a feature is ready it is deployed to a testing or staging account which should be a replica of production, this gives the DevOps team assurance that features have been testing in a realistic environment and allows them to confidently deploy any solution to production. The problem with this approach is that many teams will leave all their services running 24 hours a day, 7 days a week. Automate shutting down unrequired resources in non production environments outside of office hours, over weekends or even just holiday periods. EC2 instances, athena queries, provisioned databases, NAT gateways and many more resources all charge by time. Most services can be spun up quite quickly, and AWS Cloudformation allows developers to quickly and easily recreate entire infrastructures in a reliable and consistent state.
6. Exceeding free tier limits – Any usage above AWS’ free tier limits—or any usage after a free trial has expired—is charged at standard billing rates. To avoid these charges, make sure you’ve set up alerts that notify you before a free tier is about to expire or before you exceed what the free tier covers before incurring charges.
7. Rogue Resources – It’s common to see thousands of dollars in unattached Elastic Block Storage (EBS) volumes in AWS accounts—volumes that cost money but aren’t used for anything. To avoid seeing these charges on your bill, create a policy that automatically deletes unused EBS volumes. A similar pattern can be seen in other services, unused S3 buckets, unused secrets in Secrets Manager, Load balancers in front of unused APIs and many more. Use CostExplorer and CloudTrail to monitor what services are costing money, do regular audits of your architecture and automate the removal of unused or suspicious resources.
Also remember that if you use AWS OpsWorks to create AWS resources, you need to make sure that you also use OpsWorks to terminate those resources. If not, the OpsWorks auto-healing feature will restart them automatically, and you’ll continue to be charged
8. Excessive Backups and Encryption – we already mentioned that almost every AWS service provides some form of automated backup, versioning and encryption. Many organizations use EBS snapshots to create point-in-copy recovery points to use in case of data loss or disaster, but costs can quickly get out of control if not closely monitored.
Set a standard in your organization for how many snapshots should be retained per instance, or consider setting a policy that sends a notification or automatically deletes snapshots older than six months. Similarly look at AWS S3 lifecycles to move old archived data to cheaper storage options after a certain amount of time.
A common mistake with encryption is over-reliance of Secrets Manager, if you need to store an encrypted value but don’t require key rotation or cross account access then consider switching to use secret strings in the SSM parameter store instead.
9. Elastic IP and Carrier IP addresses – You can start to rack up charges if you assign more than one Elastic IP to the same instance, if the instance is stopped or terminated, if the IP address is unattached from the network interface, or if it’s re-mapped more than 100 times per month.
If your AWS accounts contain any disassociated Elastic IPs or Carrier IPs, be sure to either reassociate them to an instance or delete them outright in order to avoid wasted cost. For more information, see our in-depth article: Understanding Elastic IP Pricing on AWS.
10. Expiring or underutilized Reserved Instances and Savings Plans – Purchasing Reserved Instances and Savings Plans can save cloud costs substantially, but failing to utilize, monitor, or maximize them properly can lead to unexpected and unwanted charges on your monthly cloud bill. Make sure you are able to track and analyze benefits, identify waste, and allocate savings to ensure you’re optimizing your Savings Plans or Reserved Instance purchases.
The common theme for optimizing cost is the necessity to understand your infrastructure, monitor and manage access to your resources. However, as your team begins to scale cloud usage, it’s easy to exceed limits for custom metrics, alarms, dashboards, and API calls, all of which can drive up costs rapidly and unexpectedly. The answer is to spend some time investigating the right monitoring solutions and services, and automating the monitoring process to ensure consistent monitoring across all of your accounts and environments.
Conclusion
Our team offers solutions that allow businesses in a wide range of industries to unlock their potential while remaining compliant, secure and competitive. Reach out to us today to learn more about our process or start your journey.