22 ms·

A Surprise AWS Bill

240 points by oaf357 6y ago

abafazi 6y ago
> 30.6 TB?!?! how is that even possible??? Someone downloading your files from S3, that's how it's possible
RKearney 6y ago
CloudFlare's TOS is clear on using the service to serve up 13.7GB files. https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ 2.8 Limitation on Serving Non-HTML Content [...] Use of the Service for serving video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited. So 500MB limit or not, the author is already violating CloudFlare's terms of service.
- gregmac 6y ago
  Even if you read the ToS, do you really expect people to remember this? CloudFlare blends into the background when setup properly, and you don't have to think about it. If you upload a big file to your site as part of some other work -- especially personal -- most people are not running through the ToS of every service that might be involved thinking about compliance. Terns of service are also legal "CYA" documents. If CloudFlare was actually serious about that restriction, there'd be a technical limitation in place that would, for example, serve a 503.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
  - markonen 6y ago
    Cloudflare is serious about that restriction. They use it to upsell the Enterprise plan which doesn’t have it.
- syshum 6y ago
  The issue here, imo, is that CloudFlase deceptively hides these things in the TOS, and does not make it clear in the signup process, documentation, and other marketing about the limits of the service. if you look at the marketing is appears to be with out limit, now "unlimited" anything when it comes to technology should always through a red flag but CloudFlare as used this position to corner the market for CDN and DDOS services. I think it is deceptive to hide these limits of the service deep in the ToS which they know every few people actually read
  - jiofih 6y ago
    They don’t hide anything. Cloudflare has always been aimed at protecting websites and applications, not file distribution. In fact I just went to their site and could not find a mention of file hosting or even “CDN” anywhere.
    - syshum 6y ago
      Really? https://www.cloudflare.com/cdn/ https://www.cloudflare.com/cdn/ hmmm Took me 5 seconds to find it. You are referring to thier DDOS service, here we are talking about the CDN Service and clicking through about 5 pages of marketing on the CDN, including the pricing page, as well as the FAQ I see no mention of file size limits, or file type limits
prepend 6y ago
It’s always frustrated me how it’s not possible to set a quota and turn off services at quota. Logistically I know this is hard for water or power, but it should be feasible for cloud computing. But I think this is an area where it’s not in AWS’ interest to set up that kind of billing control.
- 9nGQluzmnq3M 6y ago
  Google App Engine used to do this (maybe still does?). Turns out that for most production services, having everything come to a screeching halt at the very moment your site suddenly goes viral is not desirable behavior. What you want is billing alerts, so you can be notified if usage is suddenly (say) 2x usual.
  - devwastaken 6y ago
    You don't set caps low enough to stop if you want that kind of traffic. You set them higher to the point you're not willing to pay and can't be affording.
    - 9nGQluzmnq3M 6y ago
      But with the blunt instrument of a billing quota, how do you distinguish between traffic going 100x because you're #1 on Hacker News and everybody wants to buy your product, vs somebody pwning your VMs and using them for a DDoS botnet?
      - detaro 6y ago
        It doesn't work for every use case, that doesn't make it worthless. I.e. if your homepage has (as here) 30TB traffic in a short time, that's not just a "frontpage of HN".
      - freehunter 6y ago
        Why does it have to be a blunt instrument? Why not an email saying “you’ve reached 80% of your specified quota and are about to be shut off” and if you’re #1 on HN and seeing incredible sales numbers you can just go change the quota? Or just not set a quota on your business site, but do set a quota on your personal profile page that makes you no money.
social_quotient 6y ago
AWS is an amazing service and in our past been very accommodating for surprises as long as it seems we know what we are doing and are going to mitigate the cause. I’ve rarely seen this allegiance to the “customer” by an enterprise company. They really do care and they figured out the recipe to make caring scale. What’s odd is the touch points are cold. Ticket system support, phone call back etc. it feels like it’s going to be robotic canned replies but they figured out a way to make the people on the other side smart enough to understand the issue, empowered enough to do something about it, empathetic enough to want to resolve things “fairly”.
- tehlike 6y ago
  That's from Amazon's playbook. You have an item to return, you get a refund, no fuss. Sometimes they tell you to keep the item too!
  - noir_lord 6y ago
    I once ordered an inner-tube for my roadbike, amazon sent me a box of 6, I opened an issue to get 5 returned and they just said "nah, keep em". 4 years later I've yet to buy another tube.
    - agustif 6y ago
      But surely you bought lots of other stuff from Amazon?
      - noir_lord 6y ago
        Nowhere near as much as I used to because I disagree with some of their labour practices and tax affairs. Can't fault the customer service though.
        mcherm 6y ago
        That's... really a perfect way to sum them up.
jrott 6y ago
This is the nightmare scenario with a personal AWS account. AWS billing setup makes it impossible to know that there is a giant bill until after the fact. I wish there was some way to limit the bill and just have everything shut down at that point for hobby projects.
- jimnotgym 6y ago
  This would make me move more work stuff back to AWS. I want everything to stop when I hit that limit.
  - jrott 6y ago
    Interesting I'm used to megacorp life, where the AWS bill has to be really big for people to notice. I can see how it would be helpful for startups though or for services that are providing a free tier to users.
- tylermenezes 6y ago
  Azure has configurable billing quotas. That's half of the reason I use them. (The other half being a moderately less overwhelming UI)
- smoe 6y ago
  This is why for hobby projects I try to stay away from pay per use services when possible even tough they would be cheaper. Just the peace of mind of having predictable fixed cost is worth the couple bucks I might be paying more. That said, unless that spike happens in a single day, you should be able to at least set a budget alarm to warn you. I think you should also be able to trigger shutdowns from those alerts within aws, but I never did.
- ldoughty 6y ago
  Setting up billing alarms is easy for personal accounts, it's actually only a nightmare if you go through a reseller (which is common in government contacting). The author had an incident starting June 23rd, but didn't know about it until he got his bill July 7th, that's potentially ~14 days someone could have been abusing his account. A billing alarm would have reduced this to hours or minutes. I would be interested how you suggest stopping a bill before it happens otherwise.. should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? If AWS needs to stop your billing at $10, then it might need to shut down your EC2 instance and destroy your data... If course they could offer the ability to configure the response, but if someone doesn't take the 2 minutes necessary to put a $25 billing alarm in place, what are the chances they will go through the effort of service/object based abuse policies? At the end of the day, the issue here was that the user posted something online that people could abuse.. I don't think any CDNs cover 30 TB in free tier...
9nGQluzmnq3M 6y ago
TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want pay everybody's bandwidth bills when they spin up a new instance using them. And set up billing alerts!
- curiousgal 6y ago
  I don't get why AWS refunded him, it was purely his mistake.
  - Mister_Snuggles 6y ago
    It's good customer service and leads to more profit in the long-run. If they didn't refund him, maybe he'd jump to a different provider, maybe people would see this and not sign up for AWS in the first place, etc. By refunding him, they likely keep a customer, don't scare away other potential customers, etc.
  - maest 6y ago
    Making a big deal out of this on Twitter may have something to do with it. I loathe the day when I'll have a similar problem but, as I am not a Twitter user, I won't have the luxury of getting my problems fixed.
    - nucleardog 6y ago
      AWS is actually one of the few organizations I’ve dealt with where the CSRs seem to be empowered to do stuff like this without you having to raise a storm on social media first. In my anecdotal experience, just submitting a support ticket explaining what got screwed up and you made a mistake and how you’re going to rectify it so it doesn’t happen again is usually enough to get the charges refunded. And even if you don’t submit all that, often they’ll just prompt you for it. For example, when a legacy account got compromised and rang up like $20k in charges running a bunch of servers to mine some sort of crypto, we asked for a (totally undeserved) refund and they just had us rotate all our access keys and refunded it. The support is part of the reason why I prefer dealing with AWS. I know from my time managing over a million bucks a year in AdWords spend that the reason Google’s support is non-existent isn’t because you’re not a paying customer...
deleted 6y ago
[deleted]
devwastaken 6y ago
Cloud services need to have cost caps, plain and simple. This isn't cloudlflares fault, it's Amazon's, and it's the authors. Cloudflare should be detecting overall data transfer, but there's plenty of cases where terabytes of traffic is entirely expected. We know Amazon won't fix their service, so perhaps cloudflare could impliment bandwidth limits.
- freehunter 6y ago
  100% agree that this is Amazon’s fault. Any pay-per-usage service should have a “stop service after $XX” setting.
- icefo 6y ago
  I think backblaze does it well. You can set limits for each type of API transaction and you receive email and sms notification when you get close to the cutoff limit (70% and 90% IIRC). You have a page with all the costs and how close you're to the limit if any. 2FA was also straightforward to setup. If your account is compromised the limits can still be lifted but there is limited interest in compromising a backblaze account in the first place and the 2FA makes it unlikely. I'm the sysadmin for a small nonprofit so my organization qualify for the $3500 yearly azure credits but the inability to set spending limits makes me not use azure. If I make a mistake or worse an admin account is compromised the azure bill is potentially infinite. With azure I think you can mitigate the issue somewhat by having a superAdmin account without limits and set quotas for everything else but I still don't feel at ease with that. My organization is a music festival so the infrastructure really has to work and not stop one day per year. I can keep an eye on everything during the festival and monitor spending. If things stop not during the festival people are a bit annoyed until I can look into the issue but nothing bad happens. In a perfect world I'd like to have a way to setup a limit where you need to go through support to increase it. I'd really pay for that. I think it's really too easy to receive a nasty surprise bill.
  - ghaff 6y ago
    Backblaze B2 also lets you put hard caps in place which can be nice for some situations where money is tight and the application isn't critical--and especially if you don't really have IT people monitoring.
dahdum 6y ago
From the article: > Cloudflare was the least helpful service I could have imagined given the circumstances. A long term user and on and off customer thinks they were attacked for two days and you don’t lift a finger? > File this under, “Things I should’ve known but didn’t.” Did you know that “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and Business customers and 5GB for Enterprise customers.” That’s right, Cloudflare saw requests for a 13.7 GB file and sent them straight to origin every time BY DESIGN. I don't really see how Cloudflare has much blame here. He's an "on and off customer" which I'm guessing means currently "off". They only cache a limited number of file extensions (qcow2 isn't one of them), and it's all documented. AWS always seems pretty generous in resolving these cases at least.
- ckdarby 6y ago
  I would not be surprised if AWS actually allocates marketing dollars towards covering bills like this. In the long term this is a brilliant plan because it helps prevent people from blacklisting the provider. Imagine someone gets hit with $3k bill on their personal, feels wronged, goes to work and makes effort at their employer to move off AWS. I don't know about most HN readers but I'd probably fall in this category and past places I've done work for were +$100k/month corporate bills with AWS.
  - dahdum 6y ago
    My thoughts exactly - if I felt they burned me over $2k on my personal account they'd quickly lose a couple orders of magnitude more than that on corporate.
  - khuey 6y ago
    It also doesn't cost AWS anywhere near $3k to actually move that data. Data transfer charges have to be a massive profit center for them.
    - GordonS 6y ago
      Given VPS providers give you terabytes of bandwidth included even with cheap VPS', I'd say that a $3k bandwidth bill from AWS costs AWS pretty close to $0
ta1234567890 6y ago
Off topic but on the same line. One of the most annoying things about being a consumer in the US is the ubiquitous unknown-until-last-minute pricing. You rent a car, you don't know what the total is going to be. You go to the hospital, you don't know how much you're going to have to pay. You book a hotel and don't know the total until you check out. You go to a restaurant and even if you order just one thing and saw the exact price on the menu, that's not going to be the total. You go to the grocery store, see all the prices on the items, add them up, and then when you go pay, surprise!
- _spoonman 6y ago
  Have you ever bought a house here? You go through a 1-2 month (minimum) process of providing financial documents from the past six months or more to mortgage originators, and no one can tell you how much to bring to closing until 24 hours before. It’s insane.
  - onion2k 6y ago
    It’s insane. You spelled "profitable" wrong.
  - ta1234567890 6y ago
    It is crazy. Not a house, but friend had a baby in SF/Bay Area. Neither the hospital, doctor or insurance would tell them how much the delivery would cost, not even an estimate. It took a whole year after their hospital visit to get the bill, it was $85k for a c-section and 3 days at the hospital. Fortunately their insurance took care of most of the bill. But can you imagine getting a surprise bill for $85k a year after the fact? Or not having insurance? Terrifying!
    - fmpwizard 6y ago
      We have two kids (in NC, US)), both were c-sections, we had insurance on the first one, but because she was born in Feb, we had to pay the deductible twice, because even though it was the same pregnancy, it was "over two billing years". And even with insurance we still ended up paying the rest of the bill over 3 or 4 years. Second kid was with no insurance, I think we'll finish paying that one when he turns 10 years old. I don't know about other areas in the US, but around here you can at least setup a payment plan with the hospital and they are 0% interest rate.
ricardo81 6y ago
Maybe missed it, but I didn't see which plan he was using with Cloudflare, in context of his comments about their support.
HugThem 6y ago
Summary: He published a 14GB file and one day there were 2700 downloads resulting in ~30 Terrabyte of traffic. He had the file behind CloudFlare, but since CloudFlare does not cache files larger then 512MB, all the traffic went to his S3 bucket and Amazon billed him $2700 for that.
- iakh 6y ago
  I think the most important part was left out: > AWS employee discovered that 3655 partial GETs to the object might have actually been delivered as full file requests
  - agustif 6y ago
    Wonder how many of those happen and Amazon charges their customers... HAHAHA
stefan_ 6y ago
That's a whole lot of text for "I had a 14 GiB VM image publicly linked and people discovered it", and none of it has much to do with AWS or Cloudflare. Presumably the author would do much better with a VM or something from OVH, they'll just shut you off or limit you before it becomes a problem (not that they would care about 30 TiB).
- Thaxll 6y ago
  What's more surprising is that he works at Redhat on Kubernetes it seems, rookie mistake.
  - majewsky 6y ago
    It actually makes perfect sense. As a sysadmin deploying applications on Kubernetes, you're used to have all the details of compute/network/storage hidden away from you. (Until shit hits the fan, that is.)
    - Thaxll 6y ago
      It's a linter in an IDE it's actually not type checking and it's not even close to what Go can do.
- syshum 6y ago
  I think this is less about the bill, and more about the customer service of the 2 companies, as well the "hidden" provisions in the ToS that are not clear in CloudFlare Marketing or normal setup docs.
  - blahbhthrow3748 6y ago
    The limits aren't exactly hidden in the ToS, the author just wasn't aware of them. If you don't pay money for a service what kind of support can you expect on a holiday?
    - hn_check 6y ago
saddlerustle 6y ago
Amazon gets away with high bandwidth pricing because almost all their customers are businesses with high revenue per byte served. If you want to serve large assets economically you have to look elsewhere. Bandwidth on Oracle Cloud is $0.0085/GB with the first 10TB free each month, so this would have cost only $170. Alternatively bandwidth on Backblaze B2 costs $0.01/GB, but is free out to Cloudflare, so this traffic would have been completely free.
- graton 6y ago
  Agreed. I'll be honest that I am not a fan of Oracle as a company with their actions. But I'm more than willing to take advantage of their 2 free VMs with a total of 10TB of free egress traffic.
- kitteh 6y ago
  I do recommend you spend some time looking at the quality Oracle provides. Their regions plug into transit providers (NTT, Level3, etc.) and their peering footprint is damn near non existent (they seem to be in the process of trying to fix it). Only reason I bring this up is try to send traffic to an eye ball network at peak on Oracle vs. Google/Azure/AWS at peak and you can see the difference in terms of packet loss / throughput. This is because those eyeball networks you have to be directly connected to since they run their transit hot at peak.
gchamonlive 6y ago
I think it is underestimated how complicated it is to deal with cloud services. You can do a lot with minimal training and doc reading. But these wholes in the formal understanding of how everything operates creates these kinds of vulnerabilities. Everything can have side effects in the cloud. You can set up a cheap EC2 type T feet, and without managing your cpu usage, be charged a fair amount in unlimited burst credits (which is the default for terraform for instance). You can quickly setup a WordPress instance with cloudfront and a invalidation Plug-in and be charged 6000 USD unadvertedly (https://wordpress.org/support/topic/amazon-cloudfront-invalidations/ https://wordpress.org/support/topic/amazon-cloudfront-invali...) You can set up lambda triggers and quickly do a proof of concept for an app, but forget to correctly dimension your mem usage and be charged more than you need. Cloud requires careful policy and topology consideration. There are many simple blocks that forms a complex mesh with opaque observability of potential vulnerabilities in both access and billing. Cloud is nice but it requires time and care. And with the shared responsibility model, you are responsible for that.
ykevinator 6y ago
I hate when people don't preface introductions, like "my son, max" it's so narcissistic
clarkevans 6y ago
> Now that I’m aware of the 512 MB file limit at Cloudflare, I am moving other larger files in that bucket to archive.org for now (and will add them to my supported Causes). ... > I don’t feel like archive.org should be my site’s dumping ground since it can turn a profit if it gets popular. archive.org is a stop-gap for two files for the time being. I'm trying to understand... he has decided to burden a charity with his distribution expenses?
- deleted 6y ago
  [deleted]
- throwlogon 6y ago
  There's some symmetry to the way the Internet Archive assumes consent to reproduce other people's content, and people assuming consent to take advantage of the Internet Archive's mirroring. I don't know if this is in line with the Internet Archive's terms-of-service, or the law. In this case they'd be getting stuck with a pretty big bandwidth hit.
- chmod775 6y ago
  Well. His measly 30TB probably costs the internet archive a single digit amount. He could donate $10 to the internet archive and they ought to be even. Way cheaper than those $2656.74
rytrix 6y ago
Everything is always crystal clear in hindsight, that being said I always tell my clients to set up billing alarms as one of their first tasks when getting started on AWS. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
reedwolf 6y ago
Much better incident: https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60k_on_aws_without_using_anything/ https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60...
- graton 6y ago
  The description sounds like that person is at fault. They have emails coming in for three months about thousands of dollars in charges and don't read them :(
- neurostimulant 6y ago
  It's amazing that amazon would let him run that massive instance for months without any successful payment. I would imagine other vendor would suspend the account after a few days of late payment. For example, DO was notorious on suspending and deleting data after just a few days of late payment and it caught many people off guard while on vacation (I think their policy related to late payment is much more relaxed now after it was blowing up a while ago).
Sphax 6y ago
Am I reading this wrong or the site's traffic is almost always under 1000 page views per day ? Why would you need AWS or Cloudflare to serve that ?
- blahbhthrow3748 6y ago
  Because some of the pages are 15GB!
ryanmarsh 6y ago
Saw the mention of PTSD and running towards danger and immediately thought, “oh nice hopefully I’ve found a kindred spirit”. This matters because I share little in common with my peers in this field. So I read Chris’ About page and... Do any other (combat) veterans smell something wrong with an Air Force Tech Controller (3C2X1) making statements like ”like back in the old days, when something would go bang or boom, and I’d run towards it” in a civilian venue? You know exactly what I mean, and we see it all the time. If you aren’t a veteran, especially with a job even remotely related to “running towards things that go boom” please just give us some space on this one. Thanks.
sunilkumarc 6y ago
Nicely written article. Interesting and one should definitely be aware of how certain services are charged before using them. This would be a good lesson for all of us :) On a different note, Recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner. This book covers most of the topics that you need to learn to get started: If someone is interested, here's the link: https://gumroad.com/a/238777459/MsVlG https://gumroad.com/a/238777459/MsVlG I highly recommend buying this e-book if you think AWS documentation is overwhelming.
zxcvbn4038 6y ago
The article shows again the importance of reaching out to AWS support for issues like this, they would rather have a long term customer then a one time score, they are really forgiving of one time mistakes.
afterwalk 6y ago
I wonder if aws would have given the same refund if the author wasn’t a cloud evangelist with a twitter following highly relevant to aws.
Edd314159 6y ago
I racked up a $120K+ Google Cloud bill via an unsupervised and poorly-coded script which used the geo coding API. It didn’t take much to get Google to waive it. This happens all the time I’m sure
- dathinab 6y ago
  I think so, too. But this makes it even more scary in a certain way. I mean what if they don't waive it in your case? You have no guarantees. Only the potential to cause bad press can save you. That is not a very healthy situation I believe.
ColdHeat 6y ago
DigitalOcean's S3 offering probably would've kept this bill down a bit. Probably about $300 versus $2000. https://www.digitalocean.com/docs/spaces/#bandwidth https://www.digitalocean.com/docs/spaces/#bandwidth Digital Ocean may not be the best cloud platform but it's fairly cost effective.
- andialo 6y ago
  Yes AWS is crazy expensive, their business model is simple yet super effective and they are swimming in money, power to them, make custom services with custom APIs, make it very cheap/free for low volume use and then crazy expensive when the usage goes up and the customer is vendor locked-in into their services + push enough money into marketing to so it is everyone's first choice. Not only AWS is very expensive but also rather hard to use and all their forms and services pretty difficult to navigate as well, it put me off the cloud hype for very long time until I actually discovered reasonably-priced* cloud providers like DigitalOcean (or linode, vultr,...) with also very easy to use platforms. * of course still pricier than dedicated hardware/VPS, however the premium for hourly billing and infrastructure maintenance is reasonable
jabo 6y ago
> The primary motivation was that Google had so intertwined GSuite and GCP IAM that it became overly confusing. Glad I’m not the only one confused by this.
wonderlg 6y ago
I think the main problem is that there isn’t an easy way to set a hard monthly limit on these services. I use a bunch of “freemium” services like S3 and Google Maps API and I’ve never paid a penny. I use them because they don’t cost a penny for my very limited usage, but I’m not looking forward to the day I mistakenly and disastrously exceed their free tier.
Artur96 6y ago
This is why billing alerts are mega important to setup
- Havoc 6y ago
  But wouldn't have helped much here unless you're watching your email like a hawk
- tln 6y ago
  Totally agree. I have alerts for 50% and 120%. If I get the 50% in the first week I'm digging in then. If I get the 120% I'm watching it like a hawk from then on (and letting accounting know). The article says they would not have helped but it doesn't say why... Maybe because it's delayed by a day?
  - emoprincejack 6y ago
    agreed
  - dathinab 6y ago
    They don't. But let's say you are: - on holidays - sleeping - sick - on a party - etc. Having setup alerts would still have reduced the bill. Still assuming that something like 2h pass before you can react to an email is quite reasonable which would still have been ~150$ on the big day, which is ~6x of the normal _monthly_ cost in 2h... And that is assuming the alerts are send real-time, which they are not.
- 8organicbits 6y ago
  I've found this useful to get started: https://github.com/billtrust/terraform-aws-billing-alarm https://github.com/billtrust/terraform-aws-billing-alarm
hasenpfote 6y ago
I have said it over and over and will repeat it happily: IF your Services doesn't has a proper limit, you do make yourself suddenly liable to a much higher risk than before and you have to be aware of this. It is the same shit when you rent a car: Do NEVER rent a Car without proper insurance. I'm working with GCP professionally and i have used AWS in the prev company. I do ask my manager if i can use it to try a few things out and its fine but i will not put my credit card behind an account with unlimited cost risk (its limited probably but you know what i mean). And its not even simple; Everything costs you money. Storing data, receiving data, pushing data, making api requests etc. And what i find always quite surprising: How often people, even on hn, present simple file based apis where you can upload images and edit them or upload files and download them again or offering free services and that with AWS as a backend. I just might be to long in this industry to see all those pitfalls of exploits and risks everywhere but i have the feeling that obvious respect against cloud service billing is neglected by most.
- fxtentacle 6y ago
  I believe it's the same effect as micro-transactions in mobile games. It's really easy to justify paying $1 for a small upgrade while you're playing. And only afterwards you notice that those $1 added up and have financially ruined you. In the same way, $0.08 per GB (the effective price in the article) sounds really small and easy to justify. And we forget how they can accumulate...
  - dathinab 6y ago
    Yes, but it's hard for a third party to weaponize that against you ;-) You want to destroy a small startup with a free alpha version and AWS (or similar) backing? Sure go ahead and send them tons of _legit_ (looking) traffic. This will first mess up their bill for this month and then mess up their statistics for the next month (when all the user they got disappear at once)...
    - fxtentacle 6y ago
      Agree, using cloud is kind of an invitation to your unethical competition to bankrupt you.
schmichael 6y ago
The fact that the cloud allows hobbyists, small businesses, and massive enterprises ~equal access to services is amazing. However it means sometimes things like this happen where a product’s incentives (serve any content at any cost) are wildly misaligned with a huge percentage of users needs (I’d rather my site, or preferably just the costly resource, be down than pay $2k). There’s endless tuning non-enterprises can do to get our ideal behavior: but that’s the difference between pre-cloud and post-cloud computing. It used to take monumental effort to build high scale high availability systems. Your $5/mo Dreamhost site would just die under load instead of charging you thousands. Now enterprise use cases are supported by default and it takes careful tuning to opt out.
- Slartie 6y ago
  Actually not even the typical $5/month cheap VPS offering would die under the "load" of 30TB of static content HTTP traffic being served in 2700 requests over a month. That's just a laughable drain on CPU resources that can easily be handled by even the cheapest virtual server offering, because it doesn't even get a single core of a modern CPU into the double digit utilization percentage range. The only thing that could happen is they cap your data transfer at some point. But there are cheap VPS providers out there offering several TB of gigabit speed traffic and throttling instead of a hard cap when you reach your limit.
  - schmichael 6y ago
    Ah sorry, I used Dreamhost as they were one of the biggest shared hosting providers back in the day. No VPS, just a user account on a big host. They’ve likely evolved considerably since I last used them 10+ years ago. Point being their product was targeting me and designed appropriately. I forget the details but I know there were caps that were ample for my meager needs but would prevent this sort of accidental overage. My point is that compute has become a commodity like electricity but without the built in fuses. My residential box can’t pull industrial amps.
ddevault 6y ago
This is symptomatic of hands-off ops services like AWS and CloudFlare eroding sysadmin/sysops skills in the general developer population. These problems don't go away because you pay AWS to solve them, and it'll cost you. I have very little empathy for someone who gets screwed for taking the lazy path and consolodates more and more of the internet into a handful of private megacorp's baskets. Learn how these systems work, god dammit! If you don't understand it, you're not equipped to hand control of it to a third party. Ops-as-a-service tools like AWS are highway robbery. The price is nowhere near the actual cost of providing these services - you're all being taken for a ride.
Ciantic 6y ago
This is the reason I liked Azure as I used it few years back. One could set like a prepaid plan, and there was no way to overspend it accidentally. Of course it is not ideal for companies who need their services be available for all cost, but for home users it's a nice guarantee.
namidark 6y ago
AWS bills are un-auditable. I'm convinced every org is being over-charged for bugs in their billing and tracking software. I've asked on multiple occasions where charges randomly started appearing (despite no infra changes), which weren't there the month before, and no one was able to answer on the AWS support side.
knorker 6y ago
Summary: 1. The big cloud providers charge enormously for outgoing bandwidth. Most of us know this, but unfortunately it bites people a lot. 2. If you host big files on these clouds with no limits or warnings, it's just a matter of time before this happens to you. This is why I don't run hobby things on these clouds. Any hobby project may have backends and services running on them, but NEVER anything user-accessible such a webserver, S3/GCS bucket, or similar. It's just too much of a "click here to bankrupt me". For a business it's a different matter. You are making money, and you're spending money to do so. You still need to have a DDoS plan for your outgoing traffic, but it's much easier to solve these problems if you have revenue. Revenue buys time and people.
johnklos 6y ago
Does Amazon not have... logging?
SeriousM 6y ago
The title is a clickbait.
hoppla 6y ago
When I was first looking at aws, I received a billing prediction alert that was ridiculous high. I could not find the culprit (I only had an ec2 instance and some other random services I looked into). In the end I deleted my account, to avoid this unexplainable billing. Next day, I got an email saying I erroneously got the alert. Damage was already done, but at that time, I was only exploring what the cloud had to offer, so no real damage done.... until a couple years later when I had to do some real aws for my work. Because I deleted/disabled my account, I cannot reopen an account with the same email address.
tjoff 6y ago
Can someone argue that the complexity of the cloud does not easily surpass the time and effort of just setting it up yourself? And for once all of that time and effort into setting it up is actually valuable and you get much better insights into your own operations. And the alternative is paying someone to lock you into their ecosystem. Are we really that lazy?
- dathinab 6y ago
  It's more about: - Flexible scaling. - Using other existing services from the cloud provider. - Not needing to manage any hardware. (Or to hire someone to manage the hardware for you). - Running your things in a data center. (Well connected, reasonable fault tolerant.) - Less upfront operational cost (you are not required to pay for expensive hardware upfront). For larger companies, especially such with massive amounts of micro-services it all about the flexible scaling and not needing to manage hardware.
  - tjoff 6y ago
    - usually not even needed on real hardware. Focus on a sound architecture (which will reward you in everything you do). But depending on situation VMs are an excellent choice and it is easy to spin up more - that is an antifeature - seems trivial compared to keeping up with all the cloud gotchas - possible anyway - sure, but you probably don't need expensive hardware to start, or go with VMs
    - andialo 6y ago
      Try this one: - AWS (and other cloud providers') kickbacks
      - tjoff 6y ago
        Not sure what that is? Sounds like a convoluted, time consuming scheme designed to further lock you in with the promise to recoup a fraction of AWS the tax?
logicallee 6y ago
I think this is a very common occurrence! A good alternative to this ever-present risk is to use a dedicated virtual private server that is unmetered. This would make mistakes like this (and yes, it is a mistake - it is his fault he didn't read the cloudfare details and publicly served a large VM image) impossible. Here is my referral code for the one I use[1]: https://crm.vpscheap.net/aff.php?aff=15 https://crm.vpscheap.net/aff.php?aff=15 This also (especially) applies to startups that might suddenly take off at any moment (but don't expect to.) AWS is a ticking time bomb of unexpected charges. You never know what the Internet will bring you. Go for an unmetered VPS and have 1 single well-defined charge that doesn't change. That's what I do on my side projects. [1] I previously asked Dan, the moderator here, if I can share in this way and he said it's okay. I don't have other affiliation with that company and have found it good. The last time I posted this I got 80 visitors and no complaints (and got upvotes), so I figure it is a good resource for people.
vmception 6y ago
I had a $190,000 AWS bill for an account only used for static S3 hosting. And guess what, I didn't write a blog post about it. I just went to support, said remove the charges, they identified the services that created the issue so I could kill them, and they removed the charge. Look at that, no fan fare. I had no emotion about it whatsoever. Maturity.
- jawns 6y ago
  You don't see any benefit in his sharing details about this incident in a blog post so others can avoid his mistake?
  - vmception 6y ago
    I can see some utility in writing about it. his blog post is quite literally "heart palpitations", "shock" "panic" - his words - and tweets, over a $2,000 bill. I'm not in my ideal financial circumstances, I just wouldn't have freaked out over that. I just would have handled it. And then blown them up in public forums if things didn't go my way. Customer support in the 21st century.
- stjo 6y ago
  “I had no emotion about it whatsoever. Maturity.” Doesn’t sit right with me. Wanting to share shouldn’t be condemned as immature. Besides, some of us want to read tech drama :)
  - vmception 6y ago
    The maturity aspect is knowing how to navigate a situation, not a reference to merely sharing. "heart palpitations", "shock" "panic" - his words - and tweets are not part about not knowing how to navigate a situation. I'm not in my ideal financial circumstances, I just wouldn't have freaked out over a surprise $2000 AWS bill or a $190,000 AWS bill, fully intending not to pay it if I felt my activity did not warrant that. There are no absolutes in guidances of behavior, only consequences. And the consequence here is potentially having your AWS account deleted in a month or two, with several remedies in between. > Besides, some of us want to read tech drama :) So maybe I should write a blog post about it.
Borlands 6y ago
Copying a 12GB file and sharing it publicly on S3 (how does AWS make money with S3, anyone care to answer?). I agree it’s expensive, maybe outreagously expensive. It’s great he got a refund! Not so many would be that lucky. And we can all learn
BrandoElFollito 6y ago
I asked AWS to set a limit on my spending. They said that they did not want to do that "not to break my business". I want them to - I do not care if my site is offline vs. having to pay a huge bill. That should be a choice. So I moved away from AWS. It is crazy that companies agree to such a racket (not the pricing - but the fact that you cannot set a limit). I considered to use a virtual card with a limit on it - they could not grab more than the limit and just sue me across the pound or remove my account. But I refuse to play these games with a company who does not give a shit about billing.
dvfjsdhgfv 6y ago
> Praise Twitter for at least its ability to draw attention to things. I am not sure this would’ve ended up as well as it did without it. This is another bad aspect of these stories.
tgsovlerkhgsel 6y ago
This is why I'm deathly afraid of using any major cloud provider. External traffic is effectively unlimited, and a number of possible reasons (popularity, misconfigured script pulling something in a loop, someone intentionally generating traffic to hurt me) have the possibility to throw me into arbitrary amounts of debt, with the only recourse being hope that the cloud provider will be merciful. Even if I have alerts set up: someone pulling 10 Gbit/s can generate over 100 TB per day, at $80-100 per TB. If I don't check my e-mails for weekend, I can be $30k in the hole before I notice.
pavelevst 6y ago
Good reminder to everyone including myself to use services that have budget alerts or spike protection. And to stay away from aws, even 23$ for static website hosting is a bit too much in 2020. At fairly priced hosting without fancy name, 30tb traffic can cost <10$
SergeAx 6y ago
> Moving it back to AWS from GCP bumped the AWS bill to an average of $23/month. Not too bad given the site’s traffic. I've checked the traffic, it was 2.3k users for entire June, like 75 user per day at average. It is effectively nothing, why author thinks it's okay to pay 1 cent per user per month to hosting provider? $5/mo VPS can handle two orders of magnitude more.
chmod775 6y ago
It would've literally been cheaper to burn that file to a few thousand DVDs and mail them to individual people. Nice pricing AWS.
hrmsurprises51 6y ago
> ”‘Surprise jackhole, your mortgage isn’t your most expensive bill this month, guess you shoulda enabled billing alarms!’ is crappy, broken, and WOULD NOT HAVE SOLVED THE PROBLEM!” tl;dr: Limits would have solved the problem. Alarms would have caught it sooner... The author puts a 15GB file out on the interweb and saID "I am responsible for this, use it", then fails to set limits, fails to set alarms, and doesn't want to pay the obvious and inevitable consequences. Sorry but this is a bunch of whining. They agreed to this ToS. The only thing surprising here is that Amazon was beyond courteous.
voltagex_ 6y ago
So what do you do if you want to host a file on the Internet but don't have $2000USD kicking around?
aaronchall 6y ago
How's Linode on this sort of thing?
nix23 6y ago
For a private person it's much better to use something like a private vm or a dedicated server, from Vultr or 1984hosting (Iceland) you can get a vm for just 2.50$ (only IPv6) or 5$ (IPv4 and 6) or a dedicated server from Hetzner, OVH, Scaleway (Arm64) for like 30$, some have unlimited traffic (mostly the dedicated servers) with a 1Gbit connection. NEVER use stuff you have to pay without knowing whats coming (count's for private use an small business)
rubenhak 6y ago
Not directly related to S3 traffic bill, but overall cloud cost management. Maybe some are unintentional, but still very painful. My experience with AWS & GCP. - AWS CloudWatch: expensive service, virtually unusable, hard to turn it off. - AWS overall: finding and cleaning up resources is messy. The order of creating & cleanup is not same. Closing an account is a painful process. GCP Project structure is way easier. - AWS EKS: You create a cluster, then a node group. Deleting a cluster fails if there is a node group. You go ahead to delete a node group, it complains because of "dependencies". While you're randomly looking for a "dependency" the $ clock is still ticking. You should delete the network interface before you could delete the node group, and only then the cluster. This does not sense because if the network interface was created implicitly by the node group, i should not be responsible for deleting the network interface. There should be a symmetry in create/delete operations. - GCP GKE: You create a cluster, then delete it. Cluster gets deleted - kudos, usability much better then with AWS EKS. But it turns out lots of LoadBalancers and Firewall rules are left over and still appear on the cloud bill. Those are implicitly created and should be cleaned up implicitly by GKE.

240 points by oaf357 6y ago

> 30.6 TB?!?! how is that even possible??? Someone downloading your files from S3, that's how it's possible

CloudFlare's TOS is clear on using the service to serve up 13.7GB files. https://www.cloudflare.com/terms/ https://www.cloudflare.com/terms/ 2.8 Limitation on Serving Non-HTML Content [...] Use of the Service for serving video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited. So 500MB limit or not, the author is already violating CloudFlare's terms of service.

gregmac 6y ago
Even if you read the ToS, do you really expect people to remember this? CloudFlare blends into the background when setup properly, and you don't have to think about it. If you upload a big file to your site as part of some other work -- especially personal -- most people are not running through the ToS of every service that might be involved thinking about compliance. Terns of service are also legal "CYA" documents. If CloudFlare was actually serious about that restriction, there'd be a technical limitation in place that would, for example, serve a 503.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- markonen 6y ago
  Cloudflare is serious about that restriction. They use it to upsell the Enterprise plan which doesn’t have it.
syshum 6y ago
The issue here, imo, is that CloudFlase deceptively hides these things in the TOS, and does not make it clear in the signup process, documentation, and other marketing about the limits of the service. if you look at the marketing is appears to be with out limit, now "unlimited" anything when it comes to technology should always through a red flag but CloudFlare as used this position to corner the market for CDN and DDOS services. I think it is deceptive to hide these limits of the service deep in the ToS which they know every few people actually read
- jiofih 6y ago
  They don’t hide anything. Cloudflare has always been aimed at protecting websites and applications, not file distribution. In fact I just went to their site and could not find a mention of file hosting or even “CDN” anywhere.
  - syshum 6y ago
    Really? https://www.cloudflare.com/cdn/ https://www.cloudflare.com/cdn/ hmmm Took me 5 seconds to find it. You are referring to thier DDOS service, here we are talking about the CDN Service and clicking through about 5 pages of marketing on the CDN, including the pricing page, as well as the FAQ I see no mention of file size limits, or file type limits

prepend 6y ago

It’s always frustrated me how it’s not possible to set a quota and turn off services at quota. Logistically I know this is hard for water or power, but it should be feasible for cloud computing. But I think this is an area where it’s not in AWS’ interest to set up that kind of billing control.

9nGQluzmnq3M 6y ago
Google App Engine used to do this (maybe still does?). Turns out that for most production services, having everything come to a screeching halt at the very moment your site suddenly goes viral is not desirable behavior. What you want is billing alerts, so you can be notified if usage is suddenly (say) 2x usual.
- devwastaken 6y ago
  You don't set caps low enough to stop if you want that kind of traffic. You set them higher to the point you're not willing to pay and can't be affording.
  - 9nGQluzmnq3M 6y ago
    But with the blunt instrument of a billing quota, how do you distinguish between traffic going 100x because you're #1 on Hacker News and everybody wants to buy your product, vs somebody pwning your VMs and using them for a DDoS botnet?
    - detaro 6y ago
      It doesn't work for every use case, that doesn't make it worthless. I.e. if your homepage has (as here) 30TB traffic in a short time, that's not just a "frontpage of HN".
    - freehunter 6y ago
      Why does it have to be a blunt instrument? Why not an email saying “you’ve reached 80% of your specified quota and are about to be shut off” and if you’re #1 on HN and seeing incredible sales numbers you can just go change the quota? Or just not set a quota on your business site, but do set a quota on your personal profile page that makes you no money.

social_quotient 6y ago

AWS is an amazing service and in our past been very accommodating for surprises as long as it seems we know what we are doing and are going to mitigate the cause. I’ve rarely seen this allegiance to the “customer” by an enterprise company. They really do care and they figured out the recipe to make caring scale. What’s odd is the touch points are cold. Ticket system support, phone call back etc. it feels like it’s going to be robotic canned replies but they figured out a way to make the people on the other side smart enough to understand the issue, empowered enough to do something about it, empathetic enough to want to resolve things “fairly”.

tehlike 6y ago
That's from Amazon's playbook. You have an item to return, you get a refund, no fuss. Sometimes they tell you to keep the item too!
- noir_lord 6y ago
  I once ordered an inner-tube for my roadbike, amazon sent me a box of 6, I opened an issue to get 5 returned and they just said "nah, keep em". 4 years later I've yet to buy another tube.
  - agustif 6y ago
    But surely you bought lots of other stuff from Amazon?
    - noir_lord 6y ago
      Nowhere near as much as I used to because I disagree with some of their labour practices and tax affairs. Can't fault the customer service though.
      - mcherm 6y ago
        That's... really a perfect way to sum them up.

jrott 6y ago

This is the nightmare scenario with a personal AWS account. AWS billing setup makes it impossible to know that there is a giant bill until after the fact. I wish there was some way to limit the bill and just have everything shut down at that point for hobby projects.

jimnotgym 6y ago
This would make me move more work stuff back to AWS. I want everything to stop when I hit that limit.
- jrott 6y ago
  Interesting I'm used to megacorp life, where the AWS bill has to be really big for people to notice. I can see how it would be helpful for startups though or for services that are providing a free tier to users.
tylermenezes 6y ago
Azure has configurable billing quotas. That's half of the reason I use them. (The other half being a moderately less overwhelming UI)
smoe 6y ago
This is why for hobby projects I try to stay away from pay per use services when possible even tough they would be cheaper. Just the peace of mind of having predictable fixed cost is worth the couple bucks I might be paying more. That said, unless that spike happens in a single day, you should be able to at least set a budget alarm to warn you. I think you should also be able to trigger shutdowns from those alerts within aws, but I never did.
ldoughty 6y ago
Setting up billing alarms is easy for personal accounts, it's actually only a nightmare if you go through a reseller (which is common in government contacting). The author had an incident starting June 23rd, but didn't know about it until he got his bill July 7th, that's potentially ~14 days someone could have been abusing his account. A billing alarm would have reduced this to hours or minutes. I would be interested how you suggest stopping a bill before it happens otherwise.. should AWS disable your website because you got posted on hacker news and now have a bill over your $10 limit? If AWS needs to stop your billing at $10, then it might need to shut down your EC2 instance and destroy your data... If course they could offer the ability to configure the response, but if someone doesn't take the 2 minutes necessary to put a $25 billing alarm in place, what are the chances they will go through the effort of service/object based abuse policies? At the end of the day, the issue here was that the user posted something online that people could abuse.. I don't think any CDNs cover 30 TB in free tier...

9nGQluzmnq3M 6y ago

TL;DR: Don't leave 10+ GB VM images open to the world on S3 unless you want pay everybody's bandwidth bills when they spin up a new instance using them. And set up billing alerts!

curiousgal 6y ago
I don't get why AWS refunded him, it was purely his mistake.
- Mister_Snuggles 6y ago
  It's good customer service and leads to more profit in the long-run. If they didn't refund him, maybe he'd jump to a different provider, maybe people would see this and not sign up for AWS in the first place, etc. By refunding him, they likely keep a customer, don't scare away other potential customers, etc.
- maest 6y ago
  Making a big deal out of this on Twitter may have something to do with it. I loathe the day when I'll have a similar problem but, as I am not a Twitter user, I won't have the luxury of getting my problems fixed.
  - nucleardog 6y ago
    AWS is actually one of the few organizations I’ve dealt with where the CSRs seem to be empowered to do stuff like this without you having to raise a storm on social media first. In my anecdotal experience, just submitting a support ticket explaining what got screwed up and you made a mistake and how you’re going to rectify it so it doesn’t happen again is usually enough to get the charges refunded. And even if you don’t submit all that, often they’ll just prompt you for it. For example, when a legacy account got compromised and rang up like $20k in charges running a bunch of servers to mine some sort of crypto, we asked for a (totally undeserved) refund and they just had us rotate all our access keys and refunded it. The support is part of the reason why I prefer dealing with AWS. I know from my time managing over a million bucks a year in AdWords spend that the reason Google’s support is non-existent isn’t because you’re not a paying customer...

deleted 6y ago

[deleted]

devwastaken 6y ago

Cloud services need to have cost caps, plain and simple. This isn't cloudlflares fault, it's Amazon's, and it's the authors. Cloudflare should be detecting overall data transfer, but there's plenty of cases where terabytes of traffic is entirely expected. We know Amazon won't fix their service, so perhaps cloudflare could impliment bandwidth limits.

freehunter 6y ago
100% agree that this is Amazon’s fault. Any pay-per-usage service should have a “stop service after $XX” setting.
icefo 6y ago
I think backblaze does it well. You can set limits for each type of API transaction and you receive email and sms notification when you get close to the cutoff limit (70% and 90% IIRC). You have a page with all the costs and how close you're to the limit if any. 2FA was also straightforward to setup. If your account is compromised the limits can still be lifted but there is limited interest in compromising a backblaze account in the first place and the 2FA makes it unlikely. I'm the sysadmin for a small nonprofit so my organization qualify for the $3500 yearly azure credits but the inability to set spending limits makes me not use azure. If I make a mistake or worse an admin account is compromised the azure bill is potentially infinite. With azure I think you can mitigate the issue somewhat by having a superAdmin account without limits and set quotas for everything else but I still don't feel at ease with that. My organization is a music festival so the infrastructure really has to work and not stop one day per year. I can keep an eye on everything during the festival and monitor spending. If things stop not during the festival people are a bit annoyed until I can look into the issue but nothing bad happens. In a perfect world I'd like to have a way to setup a limit where you need to go through support to increase it. I'd really pay for that. I think it's really too easy to receive a nasty surprise bill.
- ghaff 6y ago
  Backblaze B2 also lets you put hard caps in place which can be nice for some situations where money is tight and the application isn't critical--and especially if you don't really have IT people monitoring.

dahdum 6y ago

From the article: > Cloudflare was the least helpful service I could have imagined given the circumstances. A long term user and on and off customer thinks they were attacked for two days and you don’t lift a finger? > File this under, “Things I should’ve known but didn’t.” Did you know that “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and Business customers and 5GB for Enterprise customers.” That’s right, Cloudflare saw requests for a 13.7 GB file and sent them straight to origin every time BY DESIGN. I don't really see how Cloudflare has much blame here. He's an "on and off customer" which I'm guessing means currently "off". They only cache a limited number of file extensions (qcow2 isn't one of them), and it's all documented. AWS always seems pretty generous in resolving these cases at least.

ckdarby 6y ago
I would not be surprised if AWS actually allocates marketing dollars towards covering bills like this. In the long term this is a brilliant plan because it helps prevent people from blacklisting the provider. Imagine someone gets hit with $3k bill on their personal, feels wronged, goes to work and makes effort at their employer to move off AWS. I don't know about most HN readers but I'd probably fall in this category and past places I've done work for were +$100k/month corporate bills with AWS.
- dahdum 6y ago
  My thoughts exactly - if I felt they burned me over $2k on my personal account they'd quickly lose a couple orders of magnitude more than that on corporate.
- khuey 6y ago
  It also doesn't cost AWS anywhere near $3k to actually move that data. Data transfer charges have to be a massive profit center for them.
  - GordonS 6y ago
    Given VPS providers give you terabytes of bandwidth included even with cheap VPS', I'd say that a $3k bandwidth bill from AWS costs AWS pretty close to $0

ta1234567890 6y ago

Off topic but on the same line. One of the most annoying things about being a consumer in the US is the ubiquitous unknown-until-last-minute pricing. You rent a car, you don't know what the total is going to be. You go to the hospital, you don't know how much you're going to have to pay. You book a hotel and don't know the total until you check out. You go to a restaurant and even if you order just one thing and saw the exact price on the menu, that's not going to be the total. You go to the grocery store, see all the prices on the items, add them up, and then when you go pay, surprise!

_spoonman 6y ago
Have you ever bought a house here? You go through a 1-2 month (minimum) process of providing financial documents from the past six months or more to mortgage originators, and no one can tell you how much to bring to closing until 24 hours before. It’s insane.
- onion2k 6y ago
  It’s insane. You spelled "profitable" wrong.
- ta1234567890 6y ago
  It is crazy. Not a house, but friend had a baby in SF/Bay Area. Neither the hospital, doctor or insurance would tell them how much the delivery would cost, not even an estimate. It took a whole year after their hospital visit to get the bill, it was $85k for a c-section and 3 days at the hospital. Fortunately their insurance took care of most of the bill. But can you imagine getting a surprise bill for $85k a year after the fact? Or not having insurance? Terrifying!
  - fmpwizard 6y ago
    We have two kids (in NC, US)), both were c-sections, we had insurance on the first one, but because she was born in Feb, we had to pay the deductible twice, because even though it was the same pregnancy, it was "over two billing years". And even with insurance we still ended up paying the rest of the bill over 3 or 4 years. Second kid was with no insurance, I think we'll finish paying that one when he turns 10 years old. I don't know about other areas in the US, but around here you can at least setup a payment plan with the hospital and they are 0% interest rate.

ricardo81 6y ago

Maybe missed it, but I didn't see which plan he was using with Cloudflare, in context of his comments about their support.

HugThem 6y ago

Summary: He published a 14GB file and one day there were 2700 downloads resulting in ~30 Terrabyte of traffic. He had the file behind CloudFlare, but since CloudFlare does not cache files larger then 512MB, all the traffic went to his S3 bucket and Amazon billed him $2700 for that.

iakh 6y ago
I think the most important part was left out: > AWS employee discovered that 3655 partial GETs to the object might have actually been delivered as full file requests
- agustif 6y ago
  Wonder how many of those happen and Amazon charges their customers... HAHAHA

stefan_ 6y ago

That's a whole lot of text for "I had a 14 GiB VM image publicly linked and people discovered it", and none of it has much to do with AWS or Cloudflare. Presumably the author would do much better with a VM or something from OVH, they'll just shut you off or limit you before it becomes a problem (not that they would care about 30 TiB).

Thaxll 6y ago
What's more surprising is that he works at Redhat on Kubernetes it seems, rookie mistake.
- majewsky 6y ago
  It actually makes perfect sense. As a sysadmin deploying applications on Kubernetes, you're used to have all the details of compute/network/storage hidden away from you. (Until shit hits the fan, that is.)
  - Thaxll 6y ago
    It's a linter in an IDE it's actually not type checking and it's not even close to what Go can do.
syshum 6y ago
I think this is less about the bill, and more about the customer service of the 2 companies, as well the "hidden" provisions in the ToS that are not clear in CloudFlare Marketing or normal setup docs.
- blahbhthrow3748 6y ago
  The limits aren't exactly hidden in the ToS, the author just wasn't aware of them. If you don't pay money for a service what kind of support can you expect on a holiday?
  - hn_check 6y ago

saddlerustle 6y ago

Amazon gets away with high bandwidth pricing because almost all their customers are businesses with high revenue per byte served. If you want to serve large assets economically you have to look elsewhere. Bandwidth on Oracle Cloud is $0.0085/GB with the first 10TB free each month, so this would have cost only $170. Alternatively bandwidth on Backblaze B2 costs $0.01/GB, but is free out to Cloudflare, so this traffic would have been completely free.

graton 6y ago
Agreed. I'll be honest that I am not a fan of Oracle as a company with their actions. But I'm more than willing to take advantage of their 2 free VMs with a total of 10TB of free egress traffic.
kitteh 6y ago
I do recommend you spend some time looking at the quality Oracle provides. Their regions plug into transit providers (NTT, Level3, etc.) and their peering footprint is damn near non existent (they seem to be in the process of trying to fix it). Only reason I bring this up is try to send traffic to an eye ball network at peak on Oracle vs. Google/Azure/AWS at peak and you can see the difference in terms of packet loss / throughput. This is because those eyeball networks you have to be directly connected to since they run their transit hot at peak.

gchamonlive 6y ago

I think it is underestimated how complicated it is to deal with cloud services. You can do a lot with minimal training and doc reading. But these wholes in the formal understanding of how everything operates creates these kinds of vulnerabilities. Everything can have side effects in the cloud. You can set up a cheap EC2 type T feet, and without managing your cpu usage, be charged a fair amount in unlimited burst credits (which is the default for terraform for instance). You can quickly setup a WordPress instance with cloudfront and a invalidation Plug-in and be charged 6000 USD unadvertedly (https://wordpress.org/support/topic/amazon-cloudfront-invalidations/ https://wordpress.org/support/topic/amazon-cloudfront-invali...) You can set up lambda triggers and quickly do a proof of concept for an app, but forget to correctly dimension your mem usage and be charged more than you need. Cloud requires careful policy and topology consideration. There are many simple blocks that forms a complex mesh with opaque observability of potential vulnerabilities in both access and billing. Cloud is nice but it requires time and care. And with the shared responsibility model, you are responsible for that.

ykevinator 6y ago

I hate when people don't preface introductions, like "my son, max" it's so narcissistic

clarkevans 6y ago

> Now that I’m aware of the 512 MB file limit at Cloudflare, I am moving other larger files in that bucket to archive.org for now (and will add them to my supported Causes). ... > I don’t feel like archive.org should be my site’s dumping ground since it can turn a profit if it gets popular. archive.org is a stop-gap for two files for the time being. I'm trying to understand... he has decided to burden a charity with his distribution expenses?

deleted 6y ago
[deleted]
throwlogon 6y ago
There's some symmetry to the way the Internet Archive assumes consent to reproduce other people's content, and people assuming consent to take advantage of the Internet Archive's mirroring. I don't know if this is in line with the Internet Archive's terms-of-service, or the law. In this case they'd be getting stuck with a pretty big bandwidth hit.
chmod775 6y ago
Well. His measly 30TB probably costs the internet archive a single digit amount. He could donate $10 to the internet archive and they ought to be even. Way cheaper than those $2656.74

rytrix 6y ago

Everything is always crystal clear in hindsight, that being said I always tell my clients to set up billing alarms as one of their first tasks when getting started on AWS. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

reedwolf 6y ago

Much better incident: https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60k_on_aws_without_using_anything/ https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60...

graton 6y ago
The description sounds like that person is at fault. They have emails coming in for three months about thousands of dollars in charges and don't read them :(
neurostimulant 6y ago
It's amazing that amazon would let him run that massive instance for months without any successful payment. I would imagine other vendor would suspend the account after a few days of late payment. For example, DO was notorious on suspending and deleting data after just a few days of late payment and it caught many people off guard while on vacation (I think their policy related to late payment is much more relaxed now after it was blowing up a while ago).

Sphax 6y ago

Am I reading this wrong or the site's traffic is almost always under 1000 page views per day ? Why would you need AWS or Cloudflare to serve that ?

blahbhthrow3748 6y ago
Because some of the pages are 15GB!

ryanmarsh 6y ago

Saw the mention of PTSD and running towards danger and immediately thought, “oh nice hopefully I’ve found a kindred spirit”. This matters because I share little in common with my peers in this field. So I read Chris’ About page and... Do any other (combat) veterans smell something wrong with an Air Force Tech Controller (3C2X1) making statements like ”like back in the old days, when something would go bang or boom, and I’d run towards it” in a civilian venue? You know exactly what I mean, and we see it all the time. If you aren’t a veteran, especially with a job even remotely related to “running towards things that go boom” please just give us some space on this one. Thanks.

sunilkumarc 6y ago

Nicely written article. Interesting and one should definitely be aware of how certain services are charged before using them. This would be a good lesson for all of us :) On a different note, Recently I was looking to learn AWS concepts through online courses. After so much of research I finally found this e-book on Gumroad which is written by Daniel Vassallo who has worked in AWS team for 10+ years. I found this e-book very helpful as a beginner. This book covers most of the topics that you need to learn to get started: If someone is interested, here's the link: https://gumroad.com/a/238777459/MsVlG https://gumroad.com/a/238777459/MsVlG I highly recommend buying this e-book if you think AWS documentation is overwhelming.

zxcvbn4038 6y ago

The article shows again the importance of reaching out to AWS support for issues like this, they would rather have a long term customer then a one time score, they are really forgiving of one time mistakes.

afterwalk 6y ago

I wonder if aws would have given the same refund if the author wasn’t a cloud evangelist with a twitter following highly relevant to aws.

Edd314159 6y ago

I racked up a $120K+ Google Cloud bill via an unsupervised and poorly-coded script which used the geo coding API. It didn’t take much to get Google to waive it. This happens all the time I’m sure

dathinab 6y ago
I think so, too. But this makes it even more scary in a certain way. I mean what if they don't waive it in your case? You have no guarantees. Only the potential to cause bad press can save you. That is not a very healthy situation I believe.

ColdHeat 6y ago

DigitalOcean's S3 offering probably would've kept this bill down a bit. Probably about $300 versus $2000. https://www.digitalocean.com/docs/spaces/#bandwidth https://www.digitalocean.com/docs/spaces/#bandwidth Digital Ocean may not be the best cloud platform but it's fairly cost effective.

andialo 6y ago
Yes AWS is crazy expensive, their business model is simple yet super effective and they are swimming in money, power to them, make custom services with custom APIs, make it very cheap/free for low volume use and then crazy expensive when the usage goes up and the customer is vendor locked-in into their services + push enough money into marketing to so it is everyone's first choice. Not only AWS is very expensive but also rather hard to use and all their forms and services pretty difficult to navigate as well, it put me off the cloud hype for very long time until I actually discovered reasonably-priced* cloud providers like DigitalOcean (or linode, vultr,...) with also very easy to use platforms. * of course still pricier than dedicated hardware/VPS, however the premium for hourly billing and infrastructure maintenance is reasonable

jabo 6y ago

> The primary motivation was that Google had so intertwined GSuite and GCP IAM that it became overly confusing. Glad I’m not the only one confused by this.

wonderlg 6y ago

I think the main problem is that there isn’t an easy way to set a hard monthly limit on these services. I use a bunch of “freemium” services like S3 and Google Maps API and I’ve never paid a penny. I use them because they don’t cost a penny for my very limited usage, but I’m not looking forward to the day I mistakenly and disastrously exceed their free tier.

Artur96 6y ago

This is why billing alerts are mega important to setup

Havoc 6y ago
But wouldn't have helped much here unless you're watching your email like a hawk
tln 6y ago
Totally agree. I have alerts for 50% and 120%. If I get the 50% in the first week I'm digging in then. If I get the 120% I'm watching it like a hawk from then on (and letting accounting know). The article says they would not have helped but it doesn't say why... Maybe because it's delayed by a day?
- emoprincejack 6y ago
  agreed
- dathinab 6y ago
  They don't. But let's say you are: - on holidays - sleeping - sick - on a party - etc. Having setup alerts would still have reduced the bill. Still assuming that something like 2h pass before you can react to an email is quite reasonable which would still have been ~150$ on the big day, which is ~6x of the normal _monthly_ cost in 2h... And that is assuming the alerts are send real-time, which they are not.
8organicbits 6y ago
I've found this useful to get started: https://github.com/billtrust/terraform-aws-billing-alarm https://github.com/billtrust/terraform-aws-billing-alarm

hasenpfote 6y ago

I have said it over and over and will repeat it happily: IF your Services doesn't has a proper limit, you do make yourself suddenly liable to a much higher risk than before and you have to be aware of this. It is the same shit when you rent a car: Do NEVER rent a Car without proper insurance. I'm working with GCP professionally and i have used AWS in the prev company. I do ask my manager if i can use it to try a few things out and its fine but i will not put my credit card behind an account with unlimited cost risk (its limited probably but you know what i mean). And its not even simple; Everything costs you money. Storing data, receiving data, pushing data, making api requests etc. And what i find always quite surprising: How often people, even on hn, present simple file based apis where you can upload images and edit them or upload files and download them again or offering free services and that with AWS as a backend. I just might be to long in this industry to see all those pitfalls of exploits and risks everywhere but i have the feeling that obvious respect against cloud service billing is neglected by most.

fxtentacle 6y ago
I believe it's the same effect as micro-transactions in mobile games. It's really easy to justify paying $1 for a small upgrade while you're playing. And only afterwards you notice that those $1 added up and have financially ruined you. In the same way, $0.08 per GB (the effective price in the article) sounds really small and easy to justify. And we forget how they can accumulate...
- dathinab 6y ago
  Yes, but it's hard for a third party to weaponize that against you ;-) You want to destroy a small startup with a free alpha version and AWS (or similar) backing? Sure go ahead and send them tons of _legit_ (looking) traffic. This will first mess up their bill for this month and then mess up their statistics for the next month (when all the user they got disappear at once)...
  - fxtentacle 6y ago
    Agree, using cloud is kind of an invitation to your unethical competition to bankrupt you.

schmichael 6y ago

The fact that the cloud allows hobbyists, small businesses, and massive enterprises ~equal access to services is amazing. However it means sometimes things like this happen where a product’s incentives (serve any content at any cost) are wildly misaligned with a huge percentage of users needs (I’d rather my site, or preferably just the costly resource, be down than pay $2k). There’s endless tuning non-enterprises can do to get our ideal behavior: but that’s the difference between pre-cloud and post-cloud computing. It used to take monumental effort to build high scale high availability systems. Your $5/mo Dreamhost site would just die under load instead of charging you thousands. Now enterprise use cases are supported by default and it takes careful tuning to opt out.

Slartie 6y ago
Actually not even the typical $5/month cheap VPS offering would die under the "load" of 30TB of static content HTTP traffic being served in 2700 requests over a month. That's just a laughable drain on CPU resources that can easily be handled by even the cheapest virtual server offering, because it doesn't even get a single core of a modern CPU into the double digit utilization percentage range. The only thing that could happen is they cap your data transfer at some point. But there are cheap VPS providers out there offering several TB of gigabit speed traffic and throttling instead of a hard cap when you reach your limit.
- schmichael 6y ago
  Ah sorry, I used Dreamhost as they were one of the biggest shared hosting providers back in the day. No VPS, just a user account on a big host. They’ve likely evolved considerably since I last used them 10+ years ago. Point being their product was targeting me and designed appropriately. I forget the details but I know there were caps that were ample for my meager needs but would prevent this sort of accidental overage. My point is that compute has become a commodity like electricity but without the built in fuses. My residential box can’t pull industrial amps.

ddevault 6y ago

This is symptomatic of hands-off ops services like AWS and CloudFlare eroding sysadmin/sysops skills in the general developer population. These problems don't go away because you pay AWS to solve them, and it'll cost you. I have very little empathy for someone who gets screwed for taking the lazy path and consolodates more and more of the internet into a handful of private megacorp's baskets. Learn how these systems work, god dammit! If you don't understand it, you're not equipped to hand control of it to a third party. Ops-as-a-service tools like AWS are highway robbery. The price is nowhere near the actual cost of providing these services - you're all being taken for a ride.

Ciantic 6y ago

This is the reason I liked Azure as I used it few years back. One could set like a prepaid plan, and there was no way to overspend it accidentally. Of course it is not ideal for companies who need their services be available for all cost, but for home users it's a nice guarantee.

namidark 6y ago

AWS bills are un-auditable. I'm convinced every org is being over-charged for bugs in their billing and tracking software. I've asked on multiple occasions where charges randomly started appearing (despite no infra changes), which weren't there the month before, and no one was able to answer on the AWS support side.

knorker 6y ago

Summary: 1. The big cloud providers charge enormously for outgoing bandwidth. Most of us know this, but unfortunately it bites people a lot. 2. If you host big files on these clouds with no limits or warnings, it's just a matter of time before this happens to you. This is why I don't run hobby things on these clouds. Any hobby project may have backends and services running on them, but NEVER anything user-accessible such a webserver, S3/GCS bucket, or similar. It's just too much of a "click here to bankrupt me". For a business it's a different matter. You are making money, and you're spending money to do so. You still need to have a DDoS plan for your outgoing traffic, but it's much easier to solve these problems if you have revenue. Revenue buys time and people.

johnklos 6y ago

Does Amazon not have... logging?

SeriousM 6y ago

The title is a clickbait.

hoppla 6y ago

When I was first looking at aws, I received a billing prediction alert that was ridiculous high. I could not find the culprit (I only had an ec2 instance and some other random services I looked into). In the end I deleted my account, to avoid this unexplainable billing. Next day, I got an email saying I erroneously got the alert. Damage was already done, but at that time, I was only exploring what the cloud had to offer, so no real damage done.... until a couple years later when I had to do some real aws for my work. Because I deleted/disabled my account, I cannot reopen an account with the same email address.

tjoff 6y ago

Can someone argue that the complexity of the cloud does not easily surpass the time and effort of just setting it up yourself? And for once all of that time and effort into setting it up is actually valuable and you get much better insights into your own operations. And the alternative is paying someone to lock you into their ecosystem. Are we really that lazy?

dathinab 6y ago
It's more about: - Flexible scaling. - Using other existing services from the cloud provider. - Not needing to manage any hardware. (Or to hire someone to manage the hardware for you). - Running your things in a data center. (Well connected, reasonable fault tolerant.) - Less upfront operational cost (you are not required to pay for expensive hardware upfront). For larger companies, especially such with massive amounts of micro-services it all about the flexible scaling and not needing to manage hardware.
- tjoff 6y ago
  - usually not even needed on real hardware. Focus on a sound architecture (which will reward you in everything you do). But depending on situation VMs are an excellent choice and it is easy to spin up more - that is an antifeature - seems trivial compared to keeping up with all the cloud gotchas - possible anyway - sure, but you probably don't need expensive hardware to start, or go with VMs
  - andialo 6y ago
    Try this one: - AWS (and other cloud providers') kickbacks
    - tjoff 6y ago
      Not sure what that is? Sounds like a convoluted, time consuming scheme designed to further lock you in with the promise to recoup a fraction of AWS the tax?

logicallee 6y ago

I think this is a very common occurrence! A good alternative to this ever-present risk is to use a dedicated virtual private server that is unmetered. This would make mistakes like this (and yes, it is a mistake - it is his fault he didn't read the cloudfare details and publicly served a large VM image) impossible. Here is my referral code for the one I use[1]: https://crm.vpscheap.net/aff.php?aff=15 https://crm.vpscheap.net/aff.php?aff=15 This also (especially) applies to startups that might suddenly take off at any moment (but don't expect to.) AWS is a ticking time bomb of unexpected charges. You never know what the Internet will bring you. Go for an unmetered VPS and have 1 single well-defined charge that doesn't change. That's what I do on my side projects. [1] I previously asked Dan, the moderator here, if I can share in this way and he said it's okay. I don't have other affiliation with that company and have found it good. The last time I posted this I got 80 visitors and no complaints (and got upvotes), so I figure it is a good resource for people.

vmception 6y ago

I had a $190,000 AWS bill for an account only used for static S3 hosting. And guess what, I didn't write a blog post about it. I just went to support, said remove the charges, they identified the services that created the issue so I could kill them, and they removed the charge. Look at that, no fan fare. I had no emotion about it whatsoever. Maturity.

jawns 6y ago
You don't see any benefit in his sharing details about this incident in a blog post so others can avoid his mistake?
- vmception 6y ago
  I can see some utility in writing about it. his blog post is quite literally "heart palpitations", "shock" "panic" - his words - and tweets, over a $2,000 bill. I'm not in my ideal financial circumstances, I just wouldn't have freaked out over that. I just would have handled it. And then blown them up in public forums if things didn't go my way. Customer support in the 21st century.
stjo 6y ago
“I had no emotion about it whatsoever. Maturity.” Doesn’t sit right with me. Wanting to share shouldn’t be condemned as immature. Besides, some of us want to read tech drama :)
- vmception 6y ago
  The maturity aspect is knowing how to navigate a situation, not a reference to merely sharing. "heart palpitations", "shock" "panic" - his words - and tweets are not part about not knowing how to navigate a situation. I'm not in my ideal financial circumstances, I just wouldn't have freaked out over a surprise $2000 AWS bill or a $190,000 AWS bill, fully intending not to pay it if I felt my activity did not warrant that. There are no absolutes in guidances of behavior, only consequences. And the consequence here is potentially having your AWS account deleted in a month or two, with several remedies in between. > Besides, some of us want to read tech drama :) So maybe I should write a blog post about it.

Borlands 6y ago

Copying a 12GB file and sharing it publicly on S3 (how does AWS make money with S3, anyone care to answer?). I agree it’s expensive, maybe outreagously expensive. It’s great he got a refund! Not so many would be that lucky. And we can all learn

BrandoElFollito 6y ago

I asked AWS to set a limit on my spending. They said that they did not want to do that "not to break my business". I want them to - I do not care if my site is offline vs. having to pay a huge bill. That should be a choice. So I moved away from AWS. It is crazy that companies agree to such a racket (not the pricing - but the fact that you cannot set a limit). I considered to use a virtual card with a limit on it - they could not grab more than the limit and just sue me across the pound or remove my account. But I refuse to play these games with a company who does not give a shit about billing.

dvfjsdhgfv 6y ago

> Praise Twitter for at least its ability to draw attention to things. I am not sure this would’ve ended up as well as it did without it. This is another bad aspect of these stories.

tgsovlerkhgsel 6y ago

This is why I'm deathly afraid of using any major cloud provider. External traffic is effectively unlimited, and a number of possible reasons (popularity, misconfigured script pulling something in a loop, someone intentionally generating traffic to hurt me) have the possibility to throw me into arbitrary amounts of debt, with the only recourse being hope that the cloud provider will be merciful. Even if I have alerts set up: someone pulling 10 Gbit/s can generate over 100 TB per day, at $80-100 per TB. If I don't check my e-mails for weekend, I can be $30k in the hole before I notice.

pavelevst 6y ago

Good reminder to everyone including myself to use services that have budget alerts or spike protection. And to stay away from aws, even 23$ for static website hosting is a bit too much in 2020. At fairly priced hosting without fancy name, 30tb traffic can cost <10$

SergeAx 6y ago

> Moving it back to AWS from GCP bumped the AWS bill to an average of $23/month. Not too bad given the site’s traffic. I've checked the traffic, it was 2.3k users for entire June, like 75 user per day at average. It is effectively nothing, why author thinks it's okay to pay 1 cent per user per month to hosting provider? $5/mo VPS can handle two orders of magnitude more.

chmod775 6y ago

It would've literally been cheaper to burn that file to a few thousand DVDs and mail them to individual people. Nice pricing AWS.

hrmsurprises51 6y ago

> ”‘Surprise jackhole, your mortgage isn’t your most expensive bill this month, guess you shoulda enabled billing alarms!’ is crappy, broken, and WOULD NOT HAVE SOLVED THE PROBLEM!” tl;dr: Limits would have solved the problem. Alarms would have caught it sooner... The author puts a 15GB file out on the interweb and saID "I am responsible for this, use it", then fails to set limits, fails to set alarms, and doesn't want to pay the obvious and inevitable consequences. Sorry but this is a bunch of whining. They agreed to this ToS. The only thing surprising here is that Amazon was beyond courteous.

voltagex_ 6y ago

So what do you do if you want to host a file on the Internet but don't have $2000USD kicking around?

aaronchall 6y ago

How's Linode on this sort of thing?

nix23 6y ago

For a private person it's much better to use something like a private vm or a dedicated server, from Vultr or 1984hosting (Iceland) you can get a vm for just 2.50$ (only IPv6) or 5$ (IPv4 and 6) or a dedicated server from Hetzner, OVH, Scaleway (Arm64) for like 30$, some have unlimited traffic (mostly the dedicated servers) with a 1Gbit connection. NEVER use stuff you have to pay without knowing whats coming (count's for private use an small business)

rubenhak 6y ago

Not directly related to S3 traffic bill, but overall cloud cost management. Maybe some are unintentional, but still very painful. My experience with AWS & GCP. - AWS CloudWatch: expensive service, virtually unusable, hard to turn it off. - AWS overall: finding and cleaning up resources is messy. The order of creating & cleanup is not same. Closing an account is a painful process. GCP Project structure is way easier. - AWS EKS: You create a cluster, then a node group. Deleting a cluster fails if there is a node group. You go ahead to delete a node group, it complains because of "dependencies". While you're randomly looking for a "dependency" the $ clock is still ticking. You should delete the network interface before you could delete the node group, and only then the cluster. This does not sense because if the network interface was created implicitly by the node group, i should not be responsible for deleting the network interface. There should be a symmetry in create/delete operations. - GCP GKE: You create a cluster, then delete it. Cluster gets deleted - kudos, usability much better then with AWS EKS. But it turns out lots of LoadBalancers and Firewall rules are left over and still appear on the cloud bill. Those are implicitly created and should be cleaned up implicitly by GKE.