If you want to get the most out of your AWS components, you cannot ignore price. Price is a key variable for cloud optimization and without a good handle on it, your business won’t have a sustainable way to deliver fast and reliable applications to your customers.
You can find ways to deliver the fastest website or a bullet-proof architecture, and that’s OK - but if you don’t know early how much you’ll pay for it, you might put your application’s viability at risk. This is the reason Concurrency Labs’ AWS Optimization methodology revolves around Performance, Price and Availability.
In the past months I’ve been using the AWS Price List API to programmatically calculate price differences among regions. I could have used the AWS Simple Monthly Calculator, but it’s time consuming and it involves a lot of manual work, and I hate manual work. So I decided to automate things and use the Price List API in pretty much every AWS optimization project I’m involved in. I started writing a simple Python script that helped me parse JSON data from the AWS Price List API - and then my little pet project evolved into something I use very frequently.
… but wait, there’s a lot of stuff out there that calculates AWS pricing, right? Why reinvent the wheel?
First off, there are very useful features in AWS that allow you to keep track of your usage charges. You can set up CloudWatch Billing Alerts, which will send you a notification when your AWS charges exceed a specified threshold. There are also AWS Billing Reports that can be delivered to an S3 bucket of your choice, where you can analyze them or make them available for external tools to analyze.
Those are great options and I highly recommend using them. The only problem is that the information you get is delayed by several hours. CloudWatch Billing metrics are updated every 6 hours and the delivery of AWS Billing reports to your S3 bucket is kind of unpredictable. You can expect them to be updated as infrequently as once per day. When I’m running performance tests or monitoring an infrastructure update, I want to see price metrics as soon as possible and BEFORE my usage becomes a pricing issue. In many cases, waiting a full day -when I’ve already incurred in those charges- just doesn’t cut it.
A very important distinction is that all these tools tell you how much you have ALREADY spent and not how much you WILL spend given your current usage.
In addition, when I do performance tests, I like to keep all my metrics in a single place as much as possible, so I can correlate and analyze them and not have to switch consoles, tools or dashboards. CloudWatch already has a wealth of system metrics and I already have a way to export JMeter metrics to CloudWatch. The missing piece for me was to have pricing metrics in CloudWatch as well.
That’s why I wrote a tool that calculates a good approximation of what my monthly EC2 charges WILL be, in near-realtime, with a 10-minute delay. It consists of a Lambda function and a Python module that uses the AWS Price List API.
This is what the near real-time price calculation Lambda function does:
- It calculates an approximation of monthly AWS charges for all EC2 resources with a given tag, given their current usage. For example, if you tag a number of EC2 resources (instances, ELBs, volumes, etc.) with tag “stack=mywebapp”, the function will calculate charges for those resources.
- The function is called using a scheduled event configured in CloudWatch Events.
- The function takes a tag as an input parameter, for example “tag”:{“key”:“stack”, “value”:“mywebapp”}. The tag parameter is configured in the CloudWatch Events schedule.
- The function uses that tag to find corresponding AWS resources. It then reads relevant metrics from CloudWatch to calculate recent usage for each resource.
- The function projects recent usage into an approximated monthly usage and calculates price using the AWS Price List API.
- The function publishes a metric called ‘EstimatedCharges’ in CloudWatch and it includes the corresponding tag in the metric dimensions.
- The function returns a JSON object with pricing details, so it can integrate with other components, such as API Gateway.
- As of now, it supports pricing for the following AWS services: EC2, EBS, RDS, ELB, Lambda, Kinesis and Dynamo DB.
Architecture
Some limitations
Before you use this tool, you should know that:
- The only official pricing information comes from AWS - particularly the AWS Billing Reports. Anything else is just an unofficial approximation, including this function.
- There is an average 10-minute delay between usage events and price calculation. This is due to the time it takes for CloudWatch to publish metrics (5 minutes) and a wait period built into the function, to make sure all usage data has arrived to CloudWatch.
- Data Transfer cannot be extracted 100% from CloudWatch metrics alone. Data Transfer has a different price depending on whether an EC2 instance transfers data out to the internet, other EC2 instances or other AWS regions. For Data Transfer, the Lambda function considers all data as going out to the internet.
- v1 of this tool only calculates price for EC2 resources. Future versions will include more services, but will be limited to services supported by the AWS Price List API.
See it in action!
CloudFormation template
I created a CloudFormation template that deploys the Lambda function, as well as the CloudWatch Events schedule. All you have to do is specify the tag key and value you want to calculate pricing for. For example: TagKey:stack, TagValue:mywebapp
Click here to get started:
Live example
To demonstrate how the function works, I created the following configuration, which uses an Elastic Load Balancer and Auto Scaling. I ran a simple load test using Locust and gradually increased the number of simulated users for this application, to make Auto Scaling launch new EC2 instances.
My test infrastructure looks like this:
Auto Scaling eventually launched 4 t2.micro EC2 instances, which you can see in the following graph:
And here is the corresponding EstimatedCharges CloudWatch metric, which is published by the Lambda function:
If you want to know more details regarding EstimatedCharges, you can find in CloudWatch Logs the corresponding log stream for the Lambda function. The function returns a break-down of EC2 pricing items, something like this:
Adding pricing calculation for AWS resources under a different tag.
Let’s say you want this Lambda function to calculate pricing for more than one stack. In that case, all you need to do is create a new scheduled event and rule in CloudWatch Events.
From the CloudWatch console, click on Events -> Rules, then “Create Rule”
Select “Schedule” as your Event and your Lambda function as the Target. Then click on “Configure Input” -> “Constant (JSON text)” and enter a JSON parameter with the following format:
{
"tag" : {
"key" : "keyname",
"value" : "keyvalue"
}
}
Updating to the latest version
This function will be updated regularly in order to fix bugs and also to add more functionality. This means you will likely have to update the function at some point. I recommend installing the function using the CloudFormation template linked in this article, since it will simplify the update process.
To update the function, just go to the CloudFormation console, select the stack you’ve created and click on Actions -> Update Stack:
Then select “Specify an Amazon S3 template URL” and enter the following value:
http://concurrencylabs-cfn-templates.s3.amazonaws.com/lambda-near-realtime-pricing/function-plus-schedule.json
And that’s it. CloudFormation will update the function with the latest code.
GitHub repo
If you’re interested in looking at the code (and contributing!), you can find it in the following GitHub repo:
See Price Calculator code in GitHub
What’s next…
Please be reminded this tool returns a price approximation and it shouldn’t be taken as official data for price calculation. It’s very useful for testing purposes and to immediately spot sudden charges that could result in an unexpected AWS bill (instead of waiting up to 6 hours). In other words, use at your own risk. That being said, I’m planning to add support for:
- Other EC2 features, such as Reserved or Spot instances and EIPs
- Other popular AWS services supported by the Price List API, such as RDS, S3, Dynamo DB, etc.
- Other entry points, such as APIs or CLI commands
In the meanwhile, I hope you find it useful!
Update - a new version has been released that supports Serverless components - read here
