4 ms·

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds he

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs. Good on Fly.io for trying to handle such situations more sensibly.

bdcravens 2y ago
There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
- paulgb 2y ago
  I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
  - bdcravens 2y ago
    It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.
    - skeeter2020 2y ago
      this makes sense; plug the hole and stop taking on water before you ask for a free cleanup.
- perchlorate 2y ago
  You likely already know that, but to anyone else interested: a good way to prevent these kinds of situations is to run 'nosey parker' on your git repo before pushing it to a remote. It will dig through your code and configs, looking at files and through all the git history, and highlight anything that looks like tokens, passwords, keys, etc. You can set it as a pre-commit hook to block the offending code from even being committed. https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
  - Aeolun 2y ago
    Github has a similar feature that’s free for public repositories IIRC.
  - Tiberium 2y ago
    https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog is a similar tool but checks for far more secrets, so I think it'd be a better choice.
paxys 2y ago
> If the incidents that made the rounds here in the last few months are any indication They really aren't. There's a huge world out there beyond the HN front page.
AtlasBarfed 2y ago
Another reason to be cloud agnostic. They are counting that switching cost will make you eat the bill. If you have to be cloud, do dev in one cloud and test/prod in another. I know, I know, easier said than done.
- srockets 2y ago
  An argument that is often only made by folks who didn't ran anything at scale on a public cloud. If you've ever set in a room when a cloud deal was signed, you'd know it cable: you don't get to pay less by threatening to switch providers (I've seen people try to suggest that and get laughed out of the room), but by buying more from a single one. Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it. And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.
  - hinkley 2y ago
    In the IBM, HP days it wasn’t about paying less per se by threatening to switch providers, it was getting more concessions (of which money might be one). Some big companies had both systems so they could play them off of each other. Time to order more hardware, who will kiss our butts more? I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.
  - Aeolun 2y ago
    AWS is something like an insurance service. We all pay more so that they can forgive the people that mess up xD
    - srockets 2y ago
      That’s not how it works. We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances. As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.
giraffe_lady 2y ago
I have personal knowledge of two cases of this happening and both were immediately dropped. One was a high school student's personal project gone awry for like $3k worth the other was a startup where it was $120k or so. Two different providers. Neither had to particularly argue their case, just ask and wait a few days.
jmathai 2y ago
I was a PM in GCP and refund requests due to customer misconfigurations would make it to me for approval. Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary. Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.
- rawgabbit 2y ago
  So no elastic anything. Just servers?
  - jmathai 2y ago
    If you have elasticity then make sure you understand the limits. Even Serverless has these knobs but for various reasons they default to high elasticity. But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.
JimDabell 2y ago
What are you referring to, exactly? That doesn’t sound like AWS at all. AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.
- morningsam 2y ago
  You're right, I was getting two relatively recent posts mixed up: 1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1] 2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2] I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1). [1] https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776 [2] https://news.ycombinator.com/item?id=40203126 https://news.ycombinator.com/item?id=40203126
  - rmccue 2y ago
    For (2), AWS _also_ has now shipped a change that makes those requests non-billable permanently: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
ascorbic 2y ago
Well, the ones where they forgive it right away don't write blog posts about it, so you don't know about them. I guarantee that every case which makes the front page is either an example of a mistake made by an individual support engineer following the wrong script, or the person posting it is being dishonest about what actually happened (I recall a particularly suspicious recent one where they hid the name of their site). The cloud providers all have a policy of forgiving these sort of cases. The only speedbump is ascertaining if it was a genuine mistake.
turok 2y ago
We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.
- mwarkentin 2y ago
  FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-recursive-loop-detection-apis/ https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...
  - turok 2y ago
    Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.

bdcravens 2y ago

There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.

paulgb 2y ago
I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
- bdcravens 2y ago
  It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.
  - skeeter2020 2y ago
    this makes sense; plug the hole and stop taking on water before you ask for a free cleanup.
perchlorate 2y ago
You likely already know that, but to anyone else interested: a good way to prevent these kinds of situations is to run 'nosey parker' on your git repo before pushing it to a remote. It will dig through your code and configs, looking at files and through all the git history, and highlight anything that looks like tokens, passwords, keys, etc. You can set it as a pre-commit hook to block the offending code from even being committed. https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
- Aeolun 2y ago
  Github has a similar feature that’s free for public repositories IIRC.
- Tiberium 2y ago
  https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog is a similar tool but checks for far more secrets, so I think it'd be a better choice.

paxys 2y ago

> If the incidents that made the rounds here in the last few months are any indication They really aren't. There's a huge world out there beyond the HN front page.

AtlasBarfed 2y ago

Another reason to be cloud agnostic. They are counting that switching cost will make you eat the bill. If you have to be cloud, do dev in one cloud and test/prod in another. I know, I know, easier said than done.

srockets 2y ago
An argument that is often only made by folks who didn't ran anything at scale on a public cloud. If you've ever set in a room when a cloud deal was signed, you'd know it cable: you don't get to pay less by threatening to switch providers (I've seen people try to suggest that and get laughed out of the room), but by buying more from a single one. Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it. And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.
- hinkley 2y ago
  In the IBM, HP days it wasn’t about paying less per se by threatening to switch providers, it was getting more concessions (of which money might be one). Some big companies had both systems so they could play them off of each other. Time to order more hardware, who will kiss our butts more? I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.
- Aeolun 2y ago
  AWS is something like an insurance service. We all pay more so that they can forgive the people that mess up xD
  - srockets 2y ago
    That’s not how it works. We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances. As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.

giraffe_lady 2y ago

I have personal knowledge of two cases of this happening and both were immediately dropped. One was a high school student's personal project gone awry for like $3k worth the other was a startup where it was $120k or so. Two different providers. Neither had to particularly argue their case, just ask and wait a few days.

jmathai 2y ago

I was a PM in GCP and refund requests due to customer misconfigurations would make it to me for approval. Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary. Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.

rawgabbit 2y ago
So no elastic anything. Just servers?
- jmathai 2y ago
  If you have elasticity then make sure you understand the limits. Even Serverless has these knobs but for various reasons they default to high elasticity. But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.

JimDabell 2y ago

What are you referring to, exactly? That doesn’t sound like AWS at all. AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.

morningsam 2y ago
You're right, I was getting two relatively recent posts mixed up: 1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1] 2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2] I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1). [1] https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776 [2] https://news.ycombinator.com/item?id=40203126 https://news.ycombinator.com/item?id=40203126
- rmccue 2y ago
  For (2), AWS _also_ has now shipped a change that makes those requests non-billable permanently: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...

ascorbic 2y ago

Well, the ones where they forgive it right away don't write blog posts about it, so you don't know about them. I guarantee that every case which makes the front page is either an example of a mistake made by an individual support engineer following the wrong script, or the person posting it is being dishonest about what actually happened (I recall a particularly suspicious recent one where they hid the name of their site). The cloud providers all have a policy of forgiving these sort of cases. The only speedbump is ascertaining if it was a genuine mistake.

turok 2y ago

We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.

mwarkentin 2y ago
FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-recursive-loop-detection-apis/ https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...
- turok 2y ago
  Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.