4 ms·

This is only anecdotal, not personal experience, but I've read online and have had friends "oops" away large sums of money on AWS, and for the most part they se

This is only anecdotal, not personal experience, but I've read online and have had friends "oops" away large sums of money on AWS, and for the most part they seem to have at least gotten a partial discount when they contacted customer support. I strongly suspect opaque pricing and high/nonexistent limits are more about getting large organizations to transition to the cloud seamlessly (i.e. not completely caring/realizing what they're getting into for any particular migration/deployment). Tricking personal users into spending thousands by accident probably doesn't net much money compared to enterprise spend and runs the risk of alienating people who then can go into work and recommend against using a particular platform, having been burned by it on their personal accounts.

danmur 7y ago
Or just not being organised enough to calculate process realistically, that's got to be pretty difficult to do
dkdk8283 7y ago
A current coworker used to work at AWS and shared an anecdote. Makes sense but I’m surprised I haven’t seen it repeated: nobody wants to work on the billing code because it’s a mess and the penalty for a mistake is very high.
stingraycharles 7y ago
> “oops" away large sums of money on AWS, and for the most part they seem to have at least gotten a partial discount when they contacted customer support As a counter-datapoint, we accidentally left a Redshift cluster up idling for two weeks before we started getting alerts, and after numerous attempts have failed to get compensated in any way. The reasoning was that, well, it was what we requested and they had to allocate compute power to it (which we didn’t use). All in all a very frustrating experience and it makes me fairly cynical of all these “I got my money back without problems!” comments. (For what it’s worth, it was about $4k of costs which was a lot for us at the time)
- thanksforfish 7y ago
  It's also unnecessary. Give users cost controls, then you won't need the current mess of hoping support will write off $$$ of mistakes. With a risk of bankrupting a small shop if support doesn't help it drives risk averse users towards less dynamic offerings. Isn't AWS supposed to focus on the virtuous cycle of saving customers money (or at least reducing AWS supports need to write off customer mistakes)?
  - stingraycharles 7y ago
    I always have the feeling of a bit of “randomness” with these kinds of compensations. It makes sense, as it’s difficult to “codify” these types of things, lest they get abused and you might as well just lower your prices at that point. AWS is a large organization; I believe this type of stuff highly depends upon your “entrance” into the organization, i.e. the account manager. We were probably just unlucky with our Redshift troubles, but it did eventually trigger a move to Google Cloud / Bigquery, as the pay-as-you-go method seemed a bit safer (although it’s still too difficult imho to accurately estimate the costs of queries).
- mepiethree 7y ago
  similar story here, but with $80k, which almost killed the company. woot.
rozularen 7y ago
I, for one, had one of those "oops" when I misused some cloud services, making a VPS instance accessible from the network. When I got charged an extremely large amount I was contacted by the customer support. They explained to me what happened (I had leaked a SECRET in my repo) and then got refunded the total amount. It was quite an anecdotical experience because I wasn't expecting any of it as it was my mistake.
- JackFr 7y ago
  I inadvertently committed SES creds to GitHub. The only way I found out by an email from AWS telling me that the creds were suspended 24 hours later for a high reported spam rate. Someone had sent 70,000 emails (which is the default daily limit at my tier). Luckily only cost ~$8.
- scarface74 7y ago
  How do people keep doing this? It’s stated over and over again in all AWS documentation never to put access codes in code or your configuration. Locally, your keys should be stored as part of your user profile (configured with the AWS cli) and when your code is running in AWS, it should get permissions based on an IAM role attached to your EC2/ECS/Lambda.
  - applecrazy 7y ago
    My guess would be poorly configured .gitignore files. I recently had a project collaborator commit his .env file containing credentials to a MongoDB cluster because of this.
    - scarface74 7y ago
      if you follow the guidelines that are repeatedly stressed, your code should never be reading or handling AWS credentials directly. That wouldn’t be an issue. The access keys would be in your usr directory and all of the SDKs would know how to find them. When running on AWS, the SDKs get the credentials from the instance metadata.
wobbly_bush 7y ago
This has happened to me when I was a student. I didn't understand EC2 billing cycle well enough and ended up using much more than my student credits. The final bill amount was high as a student (but still less than $1000). Contacted AWS support, they waived the charges but cancelled my account and told I couldn't use that email address anymore for AWS.

danmur 7y ago

Or just not being organised enough to calculate process realistically, that's got to be pretty difficult to do

dkdk8283 7y ago

A current coworker used to work at AWS and shared an anecdote. Makes sense but I’m surprised I haven’t seen it repeated: nobody wants to work on the billing code because it’s a mess and the penalty for a mistake is very high.

stingraycharles 7y ago

> “oops" away large sums of money on AWS, and for the most part they seem to have at least gotten a partial discount when they contacted customer support As a counter-datapoint, we accidentally left a Redshift cluster up idling for two weeks before we started getting alerts, and after numerous attempts have failed to get compensated in any way. The reasoning was that, well, it was what we requested and they had to allocate compute power to it (which we didn’t use). All in all a very frustrating experience and it makes me fairly cynical of all these “I got my money back without problems!” comments. (For what it’s worth, it was about $4k of costs which was a lot for us at the time)

thanksforfish 7y ago
It's also unnecessary. Give users cost controls, then you won't need the current mess of hoping support will write off $$$ of mistakes. With a risk of bankrupting a small shop if support doesn't help it drives risk averse users towards less dynamic offerings. Isn't AWS supposed to focus on the virtuous cycle of saving customers money (or at least reducing AWS supports need to write off customer mistakes)?
- stingraycharles 7y ago
  I always have the feeling of a bit of “randomness” with these kinds of compensations. It makes sense, as it’s difficult to “codify” these types of things, lest they get abused and you might as well just lower your prices at that point. AWS is a large organization; I believe this type of stuff highly depends upon your “entrance” into the organization, i.e. the account manager. We were probably just unlucky with our Redshift troubles, but it did eventually trigger a move to Google Cloud / Bigquery, as the pay-as-you-go method seemed a bit safer (although it’s still too difficult imho to accurately estimate the costs of queries).
mepiethree 7y ago
similar story here, but with $80k, which almost killed the company. woot.

rozularen 7y ago

I, for one, had one of those "oops" when I misused some cloud services, making a VPS instance accessible from the network. When I got charged an extremely large amount I was contacted by the customer support. They explained to me what happened (I had leaked a SECRET in my repo) and then got refunded the total amount. It was quite an anecdotical experience because I wasn't expecting any of it as it was my mistake.

JackFr 7y ago
I inadvertently committed SES creds to GitHub. The only way I found out by an email from AWS telling me that the creds were suspended 24 hours later for a high reported spam rate. Someone had sent 70,000 emails (which is the default daily limit at my tier). Luckily only cost ~$8.
scarface74 7y ago
How do people keep doing this? It’s stated over and over again in all AWS documentation never to put access codes in code or your configuration. Locally, your keys should be stored as part of your user profile (configured with the AWS cli) and when your code is running in AWS, it should get permissions based on an IAM role attached to your EC2/ECS/Lambda.
- applecrazy 7y ago
  My guess would be poorly configured .gitignore files. I recently had a project collaborator commit his .env file containing credentials to a MongoDB cluster because of this.
  - scarface74 7y ago
    if you follow the guidelines that are repeatedly stressed, your code should never be reading or handling AWS credentials directly. That wouldn’t be an issue. The access keys would be in your usr directory and all of the SDKs would know how to find them. When running on AWS, the SDKs get the credentials from the instance metadata.

wobbly_bush 7y ago

This has happened to me when I was a student. I didn't understand EC2 billing cycle well enough and ended up using much more than my student credits. The final bill amount was high as a student (but still less than $1000). Contacted AWS support, they waived the charges but cancelled my account and told I couldn't use that email address anymore for AWS.