5 ms·

TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want pay everybody's bandwidth bills when they spin up a new instance using them. And se

TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want pay everybody's bandwidth bills when they spin up a new instance using them. And set up billing alerts!

curiousgal 6y ago
I don't get why AWS refunded him, it was purely his mistake.
- Mister_Snuggles 6y ago
  It's good customer service and leads to more profit in the long-run. If they didn't refund him, maybe he'd jump to a different provider, maybe people would see this and not sign up for AWS in the first place, etc. By refunding him, they likely keep a customer, don't scare away other potential customers, etc.
- maest 6y ago
  Making a big deal out of this on Twitter may have something to do with it. I loathe the day when I'll have a similar problem but, as I am not a Twitter user, I won't have the luxury of getting my problems fixed.
  - nucleardog 6y ago
    AWS is actually one of the few organizations I’ve dealt with where the CSRs seem to be empowered to do stuff like this without you having to raise a storm on social media first. In my anecdotal experience, just submitting a support ticket explaining what got screwed up and you made a mistake and how you’re going to rectify it so it doesn’t happen again is usually enough to get the charges refunded. And even if you don’t submit all that, often they’ll just prompt you for it. For example, when a legacy account got compromised and rang up like $20k in charges running a bunch of servers to mine some sort of crypto, we asked for a (totally undeserved) refund and they just had us rotate all our access keys and refunded it. The support is part of the reason why I prefer dealing with AWS. I know from my time managing over a million bucks a year in AdWords spend that the reason Google’s support is non-existent isn’t because you’re not a paying customer...
- kaetemi 6y ago
  There was apparently also a bug that caused partial http requests to be counted as complete file downloads.
- ckdarby 6y ago
  I posted elsewhere in the thread answering my theory. TL;DR Individual feels wronged, pushes things off AWS at their employer, AWS loses much more than $3k refund.
  - dathinab 6y ago
    Especially given that the 3k probably cost them much much less and it's probably also tax deductible...
- code4tee 6y ago
  Because AWS honestly cares more about having a happy loyal customer for the long term than making a few bucks in the short term. That’s why. It’s a rare mindset among large companies these days.
- detaro 6y ago
  It's a quite small sum in AWS terms, loosing a customer could easily cost them more in future revenue. And especially for bandwidth I'd assume their actual cost is way below the billed cost, so refunding it is cheap.
- deleted 6y ago
  [deleted]
- usr1106 6y ago
  He made it to the HN front page. (Maybe after the refund, but still I guess he has some track record, blog, twitter, knowing people at AWS, whatever.) I seriously doubt whether AWS would refund everybody else making stupid mistakes. If they did they would advertise it as "no billing surprises".
- iso1210 6y ago
  "A quarter-million people saw the tweet and a third of them interacted with it."
- bluedino 6y ago
  The bandwidth probably costs them $2
  - qzw 6y ago
    Exactly. It’s amazing how good customer service can be when the profit margins are high enough. It’s really only dumb companies and those with too much power that tend to combine high margins with bad customer service.
- based2 6y ago
  Marketing: https://www.yourtrainingedge.com/the-29-costly-implications-of-losing-customers-2/ https://www.yourtrainingedge.com/the-29-costly-implications-...
- dathinab 6y ago
  It's a bit skipped over but there seems to have been a bug somewhere between Cloudflare and AWS which further increased the problem. So they just waived it as it's better press. Through I'm surprised they waved all of it, I would have expected half or so to be waved but then it doesn't really matter for them. It seems to be pretty common not just for Amazon to be nice in case of such mesup's so that it is less likely to scare away people.
jon-wood 6y ago
Alternatively set the option available on S3 that pushes billing for any request and bandwidth to the requesting account: https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay...

TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want pay everybody's bandwidth bills when they spin up a new instance using them. And set up billing alerts!

curiousgal 6y ago

I don't get why AWS refunded him, it was purely his mistake.

Mister_Snuggles 6y ago
It's good customer service and leads to more profit in the long-run. If they didn't refund him, maybe he'd jump to a different provider, maybe people would see this and not sign up for AWS in the first place, etc. By refunding him, they likely keep a customer, don't scare away other potential customers, etc.
maest 6y ago
Making a big deal out of this on Twitter may have something to do with it. I loathe the day when I'll have a similar problem but, as I am not a Twitter user, I won't have the luxury of getting my problems fixed.
- nucleardog 6y ago
  AWS is actually one of the few organizations I’ve dealt with where the CSRs seem to be empowered to do stuff like this without you having to raise a storm on social media first. In my anecdotal experience, just submitting a support ticket explaining what got screwed up and you made a mistake and how you’re going to rectify it so it doesn’t happen again is usually enough to get the charges refunded. And even if you don’t submit all that, often they’ll just prompt you for it. For example, when a legacy account got compromised and rang up like $20k in charges running a bunch of servers to mine some sort of crypto, we asked for a (totally undeserved) refund and they just had us rotate all our access keys and refunded it. The support is part of the reason why I prefer dealing with AWS. I know from my time managing over a million bucks a year in AdWords spend that the reason Google’s support is non-existent isn’t because you’re not a paying customer...
kaetemi 6y ago
There was apparently also a bug that caused partial http requests to be counted as complete file downloads.
ckdarby 6y ago
I posted elsewhere in the thread answering my theory. TL;DR Individual feels wronged, pushes things off AWS at their employer, AWS loses much more than $3k refund.
- dathinab 6y ago
  Especially given that the 3k probably cost them much much less and it's probably also tax deductible...
code4tee 6y ago
Because AWS honestly cares more about having a happy loyal customer for the long term than making a few bucks in the short term. That’s why. It’s a rare mindset among large companies these days.
detaro 6y ago
It's a quite small sum in AWS terms, loosing a customer could easily cost them more in future revenue. And especially for bandwidth I'd assume their actual cost is way below the billed cost, so refunding it is cheap.
deleted 6y ago
[deleted]
usr1106 6y ago
He made it to the HN front page. (Maybe after the refund, but still I guess he has some track record, blog, twitter, knowing people at AWS, whatever.) I seriously doubt whether AWS would refund everybody else making stupid mistakes. If they did they would advertise it as "no billing surprises".
iso1210 6y ago
"A quarter-million people saw the tweet and a third of them interacted with it."
bluedino 6y ago
The bandwidth probably costs them $2
- qzw 6y ago
  Exactly. It’s amazing how good customer service can be when the profit margins are high enough. It’s really only dumb companies and those with too much power that tend to combine high margins with bad customer service.
based2 6y ago
Marketing: https://www.yourtrainingedge.com/the-29-costly-implications-of-losing-customers-2/ https://www.yourtrainingedge.com/the-29-costly-implications-...
dathinab 6y ago
It's a bit skipped over but there seems to have been a bug somewhere between Cloudflare and AWS which further increased the problem. So they just waived it as it's better press. Through I'm surprised they waved all of it, I would have expected half or so to be waved but then it doesn't really matter for them. It seems to be pretty common not just for Amazon to be nice in case of such mesup's so that it is less likely to scare away people.

jon-wood 6y ago

Alternatively set the option available on S3 that pushes billing for any request and bandwidth to the requesting account: https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPay...