9 ms·

How to See S3 Bucket Storage Types and Storage Costs

78 points by StratusBen 6y ago

joshuakelly 6y ago
It's surprising to me that cloud compute vendors make billing so complicated that a whole cottage industry of viable (?) businesses exists just to help you decipher spending.
- 0x008 6y ago
  I think that is totally on purpose. They will get you with cheap storage or instance cost and then they will drain you with egress cost or persistent storage cost or ip addresses or whatnot.
  - btmiller 6y ago
    Yup. It makes chasing the best storage deals an endless game of whack-a-mole.
    - StratusBen 6y ago
      S3 Intelligent Tiering actually does help with this considerably though. They do try to shift your objects to the storage class that makes the most sense cost-wise based off of access patterns and don't charge anything additional for doing that.
      - mbb70 6y ago
        There is a small per object/month charge for Intelligent Tiering. The 30 day minimum, even for objects in the Frequent Access Tier, can also drown you in prorated charges if applied to a high churn prefix.
- vishnugupta 6y ago
  The AWS service advisor assigned to our account would actively and frequently inform us ways of reducing the cost. Apparently one of their KPIs was cost reduction for their clients. The AWS docs may be bad at surfacing the cost information but their SAs more than filled that gap. Sure, to be assigned an SA you’ve got to be above a certain spend threshold but still it’s a pleasant experience.
systemvoltage 6y ago
Advice to clouds: Please simplify your costing structure for the small guys. It should be easy to understand and calculate. Per hour units are infuriating as well, it’s meaningless to people. If my lawyer charged me by seconds, I would have no intuitive understanding of costs until I pull out lawyercalculator.com. Billing can still happen in microunits. I’m sure they know though and this is a deliberate obfuscation technique, a dark pattern by marketing people. Edit: I want to clarify, Billing can still happen in microunits.
- unethical_ban 6y ago
  I'm with you on the first part, but not the second (ha). Time-based billing should be granular to the second/minute/hour depending on the service, but should be listed for the customer in a table upwards an order of magnitude or so. VM instances often get turned up and down in less than an hour. Serverless functions (AWS Lambda) execute and then end in milliseconds, often. But yes, billing should be simpler/more clearly reported and calculated. And many of their services seem to bill in overly complicated ways. On the other hand, I think some AWS services are essentially free for small amounts of usage, so individuals have less to worry about re: billing.
  - systemvoltage 6y ago
    > Time-based billing should be granular to the second/minute/hour depending on the service, but should be listed for the customer in a table upwards an order of magnitude or so. That's what I meant by "Billing can happen in microunits". You described it better.
- scrollaway 6y ago
  Lambda used to charge by chunks of 100ms. When they switched to charging by the millisecond, without otherwise changing the pricing, that was a universal price drop. More granularity tends to mean cheaper for you. Your lawyer doesn't charge you by the second because charging in chunks of 6 (or 15) minutes is more profitable for them. If you're a very small player you're unlikely to care about the savings in question because they'll be single-digit, if that. But I'd argue that using AWS if you're a very small player -- and don't have plans to expand -- is a poor idea. Digital Ocean, Heroku etc are far more adapted to those use cases. AWS is a poor choice at tinyscale because of how flexible it is. It's a bring-your-own-everything service. You have to set up policies in order for AWS's own services to access each other and features offered in the web UI to work properly. Their billing systems make a ton of sense for the companies that spend millions of dollars. You would not be happy doing your grocery shopping at the warehouse that provides your local shop; and yet it would probably be cheaper overall as you'd be cutting the middleman. Of course you'd run into a lot of other issues. AWS is not stopping you from using it but still. There's a reason their most successful sales program is one that can afford dumping $100k on startups: "It's not for you." (Disclaimer: There's a fuckton of awful, actual billing issues with AWS that obscure the bill for no good reason. Most small players complaining are not talking about those, aside maybe from the egregious bandwidth pricing.)
3dbrows 6y ago
Understanding cloud costs is an area Infracost (YC 2021) is working on. Give it a Terraform script and it will calculate the dollar cost; it can also diff costs in a CI pipeline: https://www.infracost.io/ https://www.infracost.io/
- leetrout 6y ago
  That’s neat. Looks like you’ve been around for a while already and are a late stage participant in YC? Love the site and the upfront screenshots.
  - aliscott 6y ago
    (co-founder here) Thanks! We started on this last August, so still pretty early and got a lot more we want to add.
- xeger 6y ago
  Infracost seems like just the thing for exploring S3 storage options before provisioning anything ... seems like it could quite powerful after the fact, too, to measure ongoing costs. As with all cost estimation tools, the projection is only as accurate as my usage projection -- but a nifty CLI and/or CI integration sure beats having to deal with AWS price sheets!
cavisne 6y ago
Worth noting that reduced redundancy costs more than standard. Amazon don’t want you to use this storage class.
- gamegoblin 6y ago
  It appears the old Reduced Redundancy storage class is on the deprecation path: https://mysteriouscode.io/blog/aws-s3-storage-classes-pricing-is-not-what-you-think/ https://mysteriouscode.io/blog/aws-s3-storage-classes-pricin... Seems to be largely replaced by One Zone - Infrequent Access https://aws.amazon.com/about-aws/whats-new/2018/04/announcing-s3-one-zone-infrequent-access-a-new-amazon-s3-storage-class/ https://aws.amazon.com/about-aws/whats-new/2018/04/announcin...
akh 6y ago
An S3 bucket has over 50 cost components, it took us a good while to map the pricing to Terraform parameters so everyone can understand that when they run `terraform apply`: https://github.com/infracost/infracost/blob/master/infracost-usage-example.yml#L152-L213 https://github.com/infracost/infracost/blob/master/infracost...

78 points by StratusBen 6y ago

It's surprising to me that cloud compute vendors make billing so complicated that a whole cottage industry of viable (?) businesses exists just to help you decipher spending.

0x008 6y ago
I think that is totally on purpose. They will get you with cheap storage or instance cost and then they will drain you with egress cost or persistent storage cost or ip addresses or whatnot.
- btmiller 6y ago
  Yup. It makes chasing the best storage deals an endless game of whack-a-mole.
  - StratusBen 6y ago
    S3 Intelligent Tiering actually does help with this considerably though. They do try to shift your objects to the storage class that makes the most sense cost-wise based off of access patterns and don't charge anything additional for doing that.
    - mbb70 6y ago
      There is a small per object/month charge for Intelligent Tiering. The 30 day minimum, even for objects in the Frequent Access Tier, can also drown you in prorated charges if applied to a high churn prefix.
vishnugupta 6y ago
The AWS service advisor assigned to our account would actively and frequently inform us ways of reducing the cost. Apparently one of their KPIs was cost reduction for their clients. The AWS docs may be bad at surfacing the cost information but their SAs more than filled that gap. Sure, to be assigned an SA you’ve got to be above a certain spend threshold but still it’s a pleasant experience.

systemvoltage 6y ago

Advice to clouds: Please simplify your costing structure for the small guys. It should be easy to understand and calculate. Per hour units are infuriating as well, it’s meaningless to people. If my lawyer charged me by seconds, I would have no intuitive understanding of costs until I pull out lawyercalculator.com. Billing can still happen in microunits. I’m sure they know though and this is a deliberate obfuscation technique, a dark pattern by marketing people. Edit: I want to clarify, Billing can still happen in microunits.

unethical_ban 6y ago
I'm with you on the first part, but not the second (ha). Time-based billing should be granular to the second/minute/hour depending on the service, but should be listed for the customer in a table upwards an order of magnitude or so. VM instances often get turned up and down in less than an hour. Serverless functions (AWS Lambda) execute and then end in milliseconds, often. But yes, billing should be simpler/more clearly reported and calculated. And many of their services seem to bill in overly complicated ways. On the other hand, I think some AWS services are essentially free for small amounts of usage, so individuals have less to worry about re: billing.
- systemvoltage 6y ago
  > Time-based billing should be granular to the second/minute/hour depending on the service, but should be listed for the customer in a table upwards an order of magnitude or so. That's what I meant by "Billing can happen in microunits". You described it better.
scrollaway 6y ago
Lambda used to charge by chunks of 100ms. When they switched to charging by the millisecond, without otherwise changing the pricing, that was a universal price drop. More granularity tends to mean cheaper for you. Your lawyer doesn't charge you by the second because charging in chunks of 6 (or 15) minutes is more profitable for them. If you're a very small player you're unlikely to care about the savings in question because they'll be single-digit, if that. But I'd argue that using AWS if you're a very small player -- and don't have plans to expand -- is a poor idea. Digital Ocean, Heroku etc are far more adapted to those use cases. AWS is a poor choice at tinyscale because of how flexible it is. It's a bring-your-own-everything service. You have to set up policies in order for AWS's own services to access each other and features offered in the web UI to work properly. Their billing systems make a ton of sense for the companies that spend millions of dollars. You would not be happy doing your grocery shopping at the warehouse that provides your local shop; and yet it would probably be cheaper overall as you'd be cutting the middleman. Of course you'd run into a lot of other issues. AWS is not stopping you from using it but still. There's a reason their most successful sales program is one that can afford dumping $100k on startups: "It's not for you." (Disclaimer: There's a fuckton of awful, actual billing issues with AWS that obscure the bill for no good reason. Most small players complaining are not talking about those, aside maybe from the egregious bandwidth pricing.)

3dbrows 6y ago

Understanding cloud costs is an area Infracost (YC 2021) is working on. Give it a Terraform script and it will calculate the dollar cost; it can also diff costs in a CI pipeline: https://www.infracost.io/ https://www.infracost.io/

leetrout 6y ago
That’s neat. Looks like you’ve been around for a while already and are a late stage participant in YC? Love the site and the upfront screenshots.
- aliscott 6y ago
  (co-founder here) Thanks! We started on this last August, so still pretty early and got a lot more we want to add.
xeger 6y ago
Infracost seems like just the thing for exploring S3 storage options before provisioning anything ... seems like it could quite powerful after the fact, too, to measure ongoing costs. As with all cost estimation tools, the projection is only as accurate as my usage projection -- but a nifty CLI and/or CI integration sure beats having to deal with AWS price sheets!

cavisne 6y ago

Worth noting that reduced redundancy costs more than standard. Amazon don’t want you to use this storage class.

gamegoblin 6y ago
It appears the old Reduced Redundancy storage class is on the deprecation path: https://mysteriouscode.io/blog/aws-s3-storage-classes-pricing-is-not-what-you-think/ https://mysteriouscode.io/blog/aws-s3-storage-classes-pricin... Seems to be largely replaced by One Zone - Infrequent Access https://aws.amazon.com/about-aws/whats-new/2018/04/announcing-s3-one-zone-infrequent-access-a-new-amazon-s3-storage-class/ https://aws.amazon.com/about-aws/whats-new/2018/04/announcin...

akh 6y ago

An S3 bucket has over 50 cost components, it took us a good while to map the pricing to Terraform parameters so everyone can understand that when they run `terraform apply`: https://github.com/infracost/infracost/blob/master/infracost-usage-example.yml#L152-L213 https://github.com/infracost/infracost/blob/master/infracost...