8 ms·

Aurora I/O optimized config saved 90% DB cost

136 points by fosterfriends 3y ago

bearjaws 3y ago
I highly recommend communicating with your business support team at AWS. Mine have always been helpful and have kept us up to date on releases like this, sometimes we even get them before they are GA.
BrentOzar 3y ago
Cut ours by over 40% too: https://www.brentozar.com/archive/2023/06/aws-aurora-cut-our-database-costs-for-sql-constantcare-again/ https://www.brentozar.com/archive/2023/06/aws-aurora-cut-our...
- fosterfriends 3y ago
  wild part is that there's been zero observable difference in performance for this config change, I think it's just a difference in billing calculation.
  - avisser 3y ago
    > I think it's just a difference in billing calculation Agree. It feels like an actuary ran some numbers and found that high I/O customers spend more on other AWS services and are more profitable than low I/0 tenants, so they changed the formula to reduce churning those high I/0 tenants.
    - jacobegold 3y ago
      This feels likely!
  - EwanToo 3y ago
    Our commit latency dropped when we turned it on, so _something_ changed on a technical level
zuckerborg0101 3y ago
I wished I knew about this!! Migrating my Aurora clusters rn
pbowyer 3y ago
Where can you see how much of your Aurora spend is on I/O? There must be a Cloudwatch metric somewhere
- fosterfriends 3y ago
  Im a big fan of Datadog's Cloud Cost management: https://www.datadoghq.com/product/cloud-cost-management/ https://www.datadoghq.com/product/cloud-cost-management/ Outside of that though, you can go to your AWS billing, filter on RDS, and slice by usage type. They make you dig, which is why I pref Datadog
- epberry 3y ago
  If you hook up https://vantage.sh https://vantage.sh you can filter by RDS costs > Cost by Category on a Cost Report to see this. Disclaimer: I work for Vantage. PS. I maintain https://ec2instances.info/rds https://ec2instances.info/rds and recently added support for Aurora I/O Optimized pricing.
- phuff 3y ago
  You can see it in cost explorer if you break down by one of the spend types. Went and checked it: group by usage type and then filter for service: RDS and you can see your io usage broken out in plain baa cost explorer.
  - pbowyer 3y ago
    Lovely, thank you!
femiagbabiaka 3y ago
Wow, good on AWS for releasing such a useful feature. It does feel a little uncomfortable to have to wait on an interested party to make optimizations to benefit you (to the tune of 90% cost savings!!), but I guess that's the name of the game in public cloud.
akokanka 3y ago
It's so amazing when CTO cares about technology and not only shareholders return and gains.
- immibis 3y ago
  You're thinking of the CEO.
- riku_iki 3y ago
  there is a chance they got pushed by large customers migrating to on prem because of high cost
jjice 3y ago
I didn't know about Aurora IO Optimized until just now. That solves my biggest fear of Aurora which is an optimized query wreaking havoc on our IO and racking up a bill. Very cool offering to see.
ignoramous 3y ago
> We reached out to some contacts at AWS to find out why the Aurora team built this. Did I/O Optimized do some clever engineering with sharding and storing data in S3? Were they just feeling generous? No surprises here. Come what may, Amazon has always strived to lower costs (the low costs, more customers, more volume fly-wheel?). This is but one example. AWS adopted cost-follow pricing for S3 (very different from value-based pricing) after apparently a lengthy debate: As they get more efficient, they want to pass down those savings to customers (as price reductions): S3 would be a tiered monthly subscription service based on average storage use, with a free tier. Customers would choose a monthly subscription rate based on how much data they typically needed to store. Simple ... The engineering team was ready to move on to the next question. Except that day we never got to the next question. We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates? How often would updates happen versus reads? ... All those factors were unknown yet could meaningfully impact our costs ... was there a way to structure our pricing [to] ensure that it would be affordable to our customers and to Amazon? ... the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. "Cost following" means that your pricing model is driven primarily by your costs, which are then passed on to your customer. This is what construction companies use, because building your customer's gazebo out of redwood will cost you a lot more than building it out of pine. If we were to use a cost-following strategy, we'd be sacrificing the simplicity of subscription pricing, but both our customers and Amazon would benefit. With cost following, whatever the developer did with S3, they would use it in a way that would meet their requirements, and they would strive to minimise their cost and, therefore, our cost too. There would be no gaming of the system, and we wouldn't have to estimate how the mythical average customer would use S3 to set our prices. From: https://archive.is/lT5zT https://archive.is/lT5zT I wonder what explains AWS' high egress costs, though.
- Nextgrid 3y ago
  > I wonder what explains AWS' high egress costs, though. Vendor lock-in. It prevents people from otherwise picking the best provider for the task at hand - for example using some managed AWS services, but keeping the bulk of your compute on-prem or at a (much cheaper) bare-metal host. It makes sense, but I wish there was an option to opt-out, as to allow high-bandwidth applications that are fully on AWS (at the moment AWS is a non-starter for many of those even if you have no intention of using AWS competitors like in the scenario above). Maybe they should just price end-user egress vs competitor egress differently (datacenter and business provider IPs are priced like now, but consumer-grade provider IPs are much cheaper/free)? That would discourage provider-hopping, while making AWS a viable provider even for high-bandwidth applications such as serving or proxying media.
nusmella 3y ago
We have a 10TB database we switched from Aurora to Postgres and it cut out bill by 80%. However, there are some differences in our schema such as now using native partitions so it's hard to tell how much $ is due to the switch and how much due to our table and query design. We have a similar story with DynamoDB too.
- thelastparadise 3y ago
  > it's hard to tell how much $ is due to the switch and how much due to our table and query design. Sounds like good material for a technical blog post.
- Ozzie_osman 3y ago
  Curious what you mean by switching from Aurora to Postgres? AWS offers Postgres on Aurora, and Postgres on regular RDS. Do you mean you switched to RDS, or off of AWS altogether, or something else?
  - ttfkam 3y ago
    Probably means Aurora MySQL. In CloudFormation and other AWS artifacts, "Aurora" is a keyword that regularly comes up meaning MySQL, since that was the original target for Aurora years before the Postgres flavor was released. There are AWS old-timers at my shop that call it Aurora, and it shows up in their YAML.
    - ttfkam 3y ago
      To whomever downvoted, when specifying the AWS::RDS::DBCluster "Engine" property in CloudFormation, aurora = mysql5.6 and below, aurora-mysql = mysql5.7 or mysql8.x, aurora-postgresql = postgres. Since 5.6 was deprecated, the "aurora" engine type was removed the CF docs, but it was there until a few years ago. "Aurora" was synonymous with MySQL for a while. #AWSHistory
      - vasco 3y ago
        People downvoted because you were assuming that when someone says "we moved to postgres" they would mean "we moved to mysql" as if they wouldn't know what they were talking about. Even your history thing makes no sense, Aurora Postgres was launched 9 months after Mysql version in July 2015.
timbaboon 3y ago
I work for a large insurance company. We had moved from on-prem to Aurora, but our design strategy did not get updated in the process, and our IO costs were about 80% of our total AWS bill. We just switched to this new IO optimised pricing and we’re seeing huge discounts. I can sleep a bit easier now :) (we still are going to change our DB design to reduce IO anyway)
thecarissa 3y ago
Greg earned a new espresso machine for his efforts here!
- fosterfriends 3y ago
  Slowly rebuilding Airbnb's Okay Coffee club...
  - jitl 3y ago
    nice
immibis 3y ago
Customer buys cloud service with unpredictable pricing. Gets bitten by high prices. Switches to (new!) cloud service with predictable pricing. Prices are lower. Also, it sounds like they could save money by not using cloud.
- Etheryte 3y ago
  Everyone who is using cloud could save money by not using cloud. However sometimes the money is worth it for the features and guarantees you get.
  - immibis 3y ago
    But especially everyone who is using cloud with unpredictable pricing models. Unpredictable means it's designed to screw you and make sure you don't have any grounds to complain.
- fosterfriends 3y ago
  For us, it would be net more expensive because we need to factor in the labor cost of an engineer maintaining a non-cloud DB.
- acdha 3y ago
  > Also, it sounds like they could save money by not using cloud. This always comes down to opportunity cost and staffing. If their team doesn’t include Postgres or MySQL experts, or security experts, etc. outsourcing that to AWS allows them to spend their staff budget on things they cannot outsource. One thing to remember is that your organization can only do so many things a year. That number varies based on complexity, internal overhead, customer and regulatory overhead, etc. but it’s finite and lower than you think. If you’re spending time on MySQL patching or trying to figure out how to patch your servers, that’s taking decisions away from your product. That can still be the right call but one thing I’ve noticed over the years is that many people are prone to dramatically underestimating that cost.
zgluck 3y ago
Does anyone have any experiences to share about the performance differences between regular RDS/MySQL and Aurora for MySQL (I/O optimized config)? (The article is about the PostgreSQL variant.)
aiunboxed 3y ago
I am an existing aurora customer and for me also aurora is already contributing to 33% of the cost, while the team is working on optimising pgsql queries, this is a great hope. I hope xmas come early for us as well :)
belter 3y ago
Maybe good to review the technical details around the feature. You might want to have a look at the blog in the last reference, and the sanity check Lambda script, before switching. Or also this specific timestamp in the presentation linked below: https://youtu.be/FBZmxLm0Bhw?t=355 https://youtu.be/FBZmxLm0Bhw?t=355 "AWS announces Amazon Aurora I/O-Optimized" - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-aurora-i-o-optimized/ https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-au... "New – Amazon Aurora I/O-Optimized Cluster Configuration with Up to 40% Cost Savings for I/O-Intensive Applications" - https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optimized-cluster-configuration-with-up-to-40-cost-savings-for-i-o-intensive-applications/ https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optim... "Getting Started with Amazon Aurora I/O-Optimized" - https://youtu.be/OlFeaVd6Ll4 https://youtu.be/OlFeaVd6Ll4 "AWS On Air ft. Introducing Amazon Aurora I/O Optimized" - https://youtu.be/FBZmxLm0Bhw https://youtu.be/FBZmxLm0Bhw "AWS Aurora IO Optimized Storage — Is it worth it?" - https://towardsaws.com/aws-aurora-io-optimized-storage-is-it-worth-it-b099dce558c8 https://towardsaws.com/aws-aurora-io-optimized-storage-is-it...

136 points by fosterfriends 3y ago

I highly recommend communicating with your business support team at AWS. Mine have always been helpful and have kept us up to date on releases like this, sometimes we even get them before they are GA.

BrentOzar 3y ago

Cut ours by over 40% too: https://www.brentozar.com/archive/2023/06/aws-aurora-cut-our-database-costs-for-sql-constantcare-again/ https://www.brentozar.com/archive/2023/06/aws-aurora-cut-our...

fosterfriends 3y ago
wild part is that there's been zero observable difference in performance for this config change, I think it's just a difference in billing calculation.
- avisser 3y ago
  > I think it's just a difference in billing calculation Agree. It feels like an actuary ran some numbers and found that high I/O customers spend more on other AWS services and are more profitable than low I/0 tenants, so they changed the formula to reduce churning those high I/0 tenants.
  - jacobegold 3y ago
    This feels likely!
- EwanToo 3y ago
  Our commit latency dropped when we turned it on, so _something_ changed on a technical level

zuckerborg0101 3y ago

I wished I knew about this!! Migrating my Aurora clusters rn

pbowyer 3y ago

Where can you see how much of your Aurora spend is on I/O? There must be a Cloudwatch metric somewhere

fosterfriends 3y ago
Im a big fan of Datadog's Cloud Cost management: https://www.datadoghq.com/product/cloud-cost-management/ https://www.datadoghq.com/product/cloud-cost-management/ Outside of that though, you can go to your AWS billing, filter on RDS, and slice by usage type. They make you dig, which is why I pref Datadog
epberry 3y ago
If you hook up https://vantage.sh https://vantage.sh you can filter by RDS costs > Cost by Category on a Cost Report to see this. Disclaimer: I work for Vantage. PS. I maintain https://ec2instances.info/rds https://ec2instances.info/rds and recently added support for Aurora I/O Optimized pricing.
phuff 3y ago
You can see it in cost explorer if you break down by one of the spend types. Went and checked it: group by usage type and then filter for service: RDS and you can see your io usage broken out in plain baa cost explorer.
- pbowyer 3y ago
  Lovely, thank you!

femiagbabiaka 3y ago

Wow, good on AWS for releasing such a useful feature. It does feel a little uncomfortable to have to wait on an interested party to make optimizations to benefit you (to the tune of 90% cost savings!!), but I guess that's the name of the game in public cloud.

akokanka 3y ago

It's so amazing when CTO cares about technology and not only shareholders return and gains.

immibis 3y ago
You're thinking of the CEO.
riku_iki 3y ago
there is a chance they got pushed by large customers migrating to on prem because of high cost

jjice 3y ago

I didn't know about Aurora IO Optimized until just now. That solves my biggest fear of Aurora which is an optimized query wreaking havoc on our IO and racking up a bill. Very cool offering to see.

ignoramous 3y ago

> We reached out to some contacts at AWS to find out why the Aurora team built this. Did I/O Optimized do some clever engineering with sharding and storing data in S3? Were they just feeling generous? No surprises here. Come what may, Amazon has always strived to lower costs (the low costs, more customers, more volume fly-wheel?). This is but one example. AWS adopted cost-follow pricing for S3 (very different from value-based pricing) after apparently a lengthy debate: As they get more efficient, they want to pass down those savings to customers (as price reductions): S3 would be a tiered monthly subscription service based on average storage use, with a free tier. Customers would choose a monthly subscription rate based on how much data they typically needed to store. Simple ... The engineering team was ready to move on to the next question. Except that day we never got to the next question. We kept discussing this question. We really did not know how developers would use S3 when it launched. Would they store mostly large objects with low retrieval rates? Small objects with high retrieval rates? How often would updates happen versus reads? ... All those factors were unknown yet could meaningfully impact our costs ... was there a way to structure our pricing [to] ensure that it would be affordable to our customers and to Amazon? ... the discussion moved away from a tiered subscription pricing strategy and toward a cost-following strategy. "Cost following" means that your pricing model is driven primarily by your costs, which are then passed on to your customer. This is what construction companies use, because building your customer's gazebo out of redwood will cost you a lot more than building it out of pine. If we were to use a cost-following strategy, we'd be sacrificing the simplicity of subscription pricing, but both our customers and Amazon would benefit. With cost following, whatever the developer did with S3, they would use it in a way that would meet their requirements, and they would strive to minimise their cost and, therefore, our cost too. There would be no gaming of the system, and we wouldn't have to estimate how the mythical average customer would use S3 to set our prices. From: https://archive.is/lT5zT https://archive.is/lT5zT I wonder what explains AWS' high egress costs, though.

Nextgrid 3y ago
> I wonder what explains AWS' high egress costs, though. Vendor lock-in. It prevents people from otherwise picking the best provider for the task at hand - for example using some managed AWS services, but keeping the bulk of your compute on-prem or at a (much cheaper) bare-metal host. It makes sense, but I wish there was an option to opt-out, as to allow high-bandwidth applications that are fully on AWS (at the moment AWS is a non-starter for many of those even if you have no intention of using AWS competitors like in the scenario above). Maybe they should just price end-user egress vs competitor egress differently (datacenter and business provider IPs are priced like now, but consumer-grade provider IPs are much cheaper/free)? That would discourage provider-hopping, while making AWS a viable provider even for high-bandwidth applications such as serving or proxying media.

nusmella 3y ago

We have a 10TB database we switched from Aurora to Postgres and it cut out bill by 80%. However, there are some differences in our schema such as now using native partitions so it's hard to tell how much $ is due to the switch and how much due to our table and query design. We have a similar story with DynamoDB too.

thelastparadise 3y ago
> it's hard to tell how much $ is due to the switch and how much due to our table and query design. Sounds like good material for a technical blog post.
Ozzie_osman 3y ago
Curious what you mean by switching from Aurora to Postgres? AWS offers Postgres on Aurora, and Postgres on regular RDS. Do you mean you switched to RDS, or off of AWS altogether, or something else?
- ttfkam 3y ago
  Probably means Aurora MySQL. In CloudFormation and other AWS artifacts, "Aurora" is a keyword that regularly comes up meaning MySQL, since that was the original target for Aurora years before the Postgres flavor was released. There are AWS old-timers at my shop that call it Aurora, and it shows up in their YAML.
  - ttfkam 3y ago
    To whomever downvoted, when specifying the AWS::RDS::DBCluster "Engine" property in CloudFormation, aurora = mysql5.6 and below, aurora-mysql = mysql5.7 or mysql8.x, aurora-postgresql = postgres. Since 5.6 was deprecated, the "aurora" engine type was removed the CF docs, but it was there until a few years ago. "Aurora" was synonymous with MySQL for a while. #AWSHistory
    - vasco 3y ago
      People downvoted because you were assuming that when someone says "we moved to postgres" they would mean "we moved to mysql" as if they wouldn't know what they were talking about. Even your history thing makes no sense, Aurora Postgres was launched 9 months after Mysql version in July 2015.

timbaboon 3y ago

I work for a large insurance company. We had moved from on-prem to Aurora, but our design strategy did not get updated in the process, and our IO costs were about 80% of our total AWS bill. We just switched to this new IO optimised pricing and we’re seeing huge discounts. I can sleep a bit easier now :) (we still are going to change our DB design to reduce IO anyway)

thecarissa 3y ago

Greg earned a new espresso machine for his efforts here!

fosterfriends 3y ago
Slowly rebuilding Airbnb's Okay Coffee club...
- jitl 3y ago
  nice

immibis 3y ago

Customer buys cloud service with unpredictable pricing. Gets bitten by high prices. Switches to (new!) cloud service with predictable pricing. Prices are lower. Also, it sounds like they could save money by not using cloud.

Etheryte 3y ago
Everyone who is using cloud could save money by not using cloud. However sometimes the money is worth it for the features and guarantees you get.
- immibis 3y ago
  But especially everyone who is using cloud with unpredictable pricing models. Unpredictable means it's designed to screw you and make sure you don't have any grounds to complain.
fosterfriends 3y ago
For us, it would be net more expensive because we need to factor in the labor cost of an engineer maintaining a non-cloud DB.
acdha 3y ago
> Also, it sounds like they could save money by not using cloud. This always comes down to opportunity cost and staffing. If their team doesn’t include Postgres or MySQL experts, or security experts, etc. outsourcing that to AWS allows them to spend their staff budget on things they cannot outsource. One thing to remember is that your organization can only do so many things a year. That number varies based on complexity, internal overhead, customer and regulatory overhead, etc. but it’s finite and lower than you think. If you’re spending time on MySQL patching or trying to figure out how to patch your servers, that’s taking decisions away from your product. That can still be the right call but one thing I’ve noticed over the years is that many people are prone to dramatically underestimating that cost.

zgluck 3y ago

Does anyone have any experiences to share about the performance differences between regular RDS/MySQL and Aurora for MySQL (I/O optimized config)? (The article is about the PostgreSQL variant.)

aiunboxed 3y ago

I am an existing aurora customer and for me also aurora is already contributing to 33% of the cost, while the team is working on optimising pgsql queries, this is a great hope. I hope xmas come early for us as well :)

belter 3y ago

Maybe good to review the technical details around the feature. You might want to have a look at the blog in the last reference, and the sanity check Lambda script, before switching. Or also this specific timestamp in the presentation linked below: https://youtu.be/FBZmxLm0Bhw?t=355 https://youtu.be/FBZmxLm0Bhw?t=355 "AWS announces Amazon Aurora I/O-Optimized" - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-aurora-i-o-optimized/ https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-au... "New – Amazon Aurora I/O-Optimized Cluster Configuration with Up to 40% Cost Savings for I/O-Intensive Applications" - https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optimized-cluster-configuration-with-up-to-40-cost-savings-for-i-o-intensive-applications/ https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optim... "Getting Started with Amazon Aurora I/O-Optimized" - https://youtu.be/OlFeaVd6Ll4 https://youtu.be/OlFeaVd6Ll4 "AWS On Air ft. Introducing Amazon Aurora I/O Optimized" - https://youtu.be/FBZmxLm0Bhw https://youtu.be/FBZmxLm0Bhw "AWS Aurora IO Optimized Storage — Is it worth it?" - https://towardsaws.com/aws-aurora-io-optimized-storage-is-it-worth-it-b099dce558c8 https://towardsaws.com/aws-aurora-io-optimized-storage-is-it...