18 ms·
A few years ago my startup was killed by a AWS mistake that ran overnight. The irony: my AWS expert at the time had made exactly the same provisioning mistake a
by appstorelottery 7y ago
A few years ago my startup was killed by a AWS mistake that ran overnight. The irony: my AWS expert at the time had made exactly the same provisioning mistake at his previous job - so I figured he'd never make a $80k mistake again. It turns out - his mistake with my startup was even more impressive. More positively - he did help shell out with me to cover the cost & overnight we were out of money. The mistake shocked me so much, and I've since heard so many stories of similar mistakes. The event hit me so hard I went back in time to PHP and shared hosting. Not kidding.
- jacobra2 7y agoWhat was the mistake?
- appstorelottery 7y agoRunning up a shitload of instances for testing and leaving all of them running overnight. Each of these instances continually rendered 4k video data to storage. This kind of test was supposed to be 1000x smaller, running for at most 10-20 seconds at time. He had written his own provisioning system which - according to his report - failed to properly manage instances "weird" edge case. No kidding.
- appstorelottery 7y agoEvery morning I would check AWS billing just out of habit. I'm just thankful I did - otherwise everything would have kept running... The lesson for me was don't trust your internally-hacked-together instance management system. The AWS interface to storage and instances is the base truth. And perhaps more importantly - I'm never getting into another startup which has financial risk like that without being a core expert in that risk/tech. I was focused on the business + client code - and had very little clue about the nitty-gritty of AWS. I should have been more involved with the code on that side, or at least the data-flow architecture.
- seibelj 7y agoAssuming you were incorporated and had a business account - declare bankruptcy and the bill goes away. I don’t understand why you would still pay the bill if you were going out of business anyway.
- appstorelottery 7y agoWhy didn't I file bankruptcy? This happened in Australia and declaring bankruptcy was not the right thing to do - for many reasons, not the least of which it makes it much harder to operate as a director of a previously bankrupt company, but in the worst case my bank would have just gone after me as I'd given a personal guarantee.
- garmaine 7y agoThere is no concept of limited liability in Australia?
- jkaplowitz 7y agoEven in the United States, most small business loans require personal guarantees which narrowly override the corporate limited liability to make that guarantor liable for that debt if the company doesn't pay. There are some rare exceptions, and possibly more for startups funded by big-name VCs, but I don't know.
- Scoundreller 7y agoBut this isn't a small business loan: it's a debt to Amazon.
- namdnay 7y agoExcept the loan money will go straight to Amazon, and you are now unable to repay the loan to the bank
- yingw787 7y agoHoly crap dude, that's some nightmare shit right there. Does AWS update the billing console per day or upon request? I get charged per month, but I should add a habit in my habit tracker to learn more about my expenses...
- freeone3000 7y agoHourly. You can also set up billing alerts, which will email you.
- lostlogin 7y agoThis is what was needed.
- WrtCdEvrydy 7y agoBe aware that some services bill asynchronously so it can take 24 hours in some instances.
- user5994461 7y agoHow many is a shitload of instances? Are we talking tens, hundreds, thousands? In my experience AWS had very stringent limits on the amount of active instances of each type (starts around 10 for new accounts, 2 for the more expensive instances). It takes tickets to support then days of waiting to raise these limits. That should have prevented your company from creating tens of instances, let alone hundreds, unless that's already your typical daily usage.
- redis_mlc 7y agoThere used to be no limit on EC2 instances.
- deleted 7y ago[deleted]
- x86_64Ubuntu 7y agoWhat's the technical process to ensure that this never happens? Nowadays, having to have someone "watch" the test and then kill the instances is manual labor which is a no-no. So how do you make it so that your test fires up the instances, and then kills them when the test is done.
- dev_throw 7y agoYou can use auto scaling groups with a load balancer to terminate instances when not in use and spin them up as required.
- rcxdude 7y agoYou can do timed instances, and/or make the instances have timed job to shutdown after a fixed time (which is what I use to shut down an instance which only gets spooled up for occasional CI jobs after an hour).
- bluecmd 7y ago+1. When I had to use AWS for batch workloads, which at the time at least didn't have a TTL attribute on VMs, I made sure that the VM first scheduled a shutdown in like 30 min if the test was supposed to only run in 10 min.
- Twirrim 7y agoNot sure if this would help in this particular scenario, but unit and integration testing of operations scripts can save a lot of pain, anguish and $$s too. It's horrifying how many places treat writing tests for services as critical, but then completely fail to write tests for their operational tooling. Including tools responsible for scaling up and down infrastructure, deleting objects etc.
- dirtydroog 7y agoBut if a test fails does it now mean you're bankrupt?
- tomerico 7y agoWhy is there no way to set a limit on billing on AWS? Especially for cases like this, where killing testing instances does not have a dramatic negative effect...
- strongbond 7y agoAgreed. The simple solution is an expenditure cap. Why can't Amazon implement one? The fear of it going wrong like this would make me keep away from AWS forever.
- joshvm 7y agoNowadays quotas give you some safety net. For example you usually have to request more than one GPU to avoid burning money that way, or more than say 32 instances. It should not be possible for a new account to spawn 1k VMs overnight. The problem with billing is that often these charges are not calculated instantly, and others are not trivial to deal with. For example what happens if you go over budget on bandwidth or bucket storage, but still within quota? What do you kill? Do you immediately shut down everything? Do you lose data? There are lots of edge cases. You can normally write your own hooks to monitor billing alerts and take action appropriately.
- ehsankia 7y agoWait, is there really not one on AWS? I thought this was the #1 most important feature on any such cloud systems. It's the very very first thing I set when setting up my GCloud hobby project. I was like, this is fun and all, but I don't care about this enough so I limited it to 3$ per day and 50$ per month. If it goes above, I'm very happy to let it die, and it also gives me a warning so I know something is up. The 2 times it triggered, there was something I managed to fix so the tool is still up and running costing pennies.
- WrtCdEvrydy 7y agoThis is why it's Terraform or nothing for me.
- scubbo 7y agoI'd be fascinated to hear how Terraform would have intelligently known that those instances were not meant to stay on overnight.
- WrtCdEvrydy 7y agoHonestly, I'd create the instances using an ASG, then set the ASG size to 0 (or throw inside a while loop until any errors go away). Always create instances from an AMI and always put them in an ASG (even if the ASG only has 1 item min, target, and max on it).
- jenkinstrigger 7y agoI love Terraform and ASGs but that still doesn't solve the fact that their SRE overprovisioned. They might have even used both things!
- deleted 7y ago[deleted]
- smiths1999 7y agoThis has happened to me several times, albeit at a much smaller scale. I fire up a few GPU instances for training neural networks and when I got to shut the instances down I forget that you always need to refresh the instance page before telling AWS to stop my instances. I still go through all the confirmations saying I do, indeed, want to stop all instances. However, these few times I forgot to refresh to make sure they actually were shutting down and simply went to bed. Not an $80k mistake, but certainly a couple hundred dollars, which hurts as a grad student. Now I have learned, _always_ refresh the page and instance list prior to shutting anything down and _always_ confirm the shutdown was successful.
- quickthrower2 7y agoUsing a post paid service and getting bill shock.
- BluePen7 7y agoNot who you asked, but my mistake was transferring an S3 bucket full of unused old customer web assets to glacier, we were paying a lot to host them each month, and weren't using them anymore. I set the lifecycle rule on all objects in the bucket, for as soon as possible (24 hours). About 2 days later first thing in the morning I get a bunch of frantic messages from my manager that whatever script I was running, please stop it, before I'd even done anything for the day. The lifecycle rule had taken effect near the end of the previous day, and he was just getting all the billing alerts from overnight, it was all done. I read about glacier pricing, but didn't realize there was a lifecycle transfer fee per 1000 objects (I forget the exact price, maybe $0.05 per 1000 objects). That section was a lot further down the pricing page. The bucket contained over 700 million small files. I'd just blown $42,000. That was over a month's AWS budget for us, in the end AWS gave us 10% back. On the plus side, I didn't get in too much trouble, and given we'd break even in 4 years on S3 costs, upper management was gracious enough to see it as an unplanned investment. TLDR: My company spent 42k for me to learn to read to the bottom of every AWS pricing page.
- Nition 7y agoWhat would have been the correct solution here? Group them into compressed archives first to reduce file count?
- jenkinstrigger 7y agoOne .zip to rule them all :)
- Nition 7y agoHaha, I original wrote "one giant zip file?" but I decided to rephrase it as a more serious answer.
- stainforth 7y agoWhy would they create a pricing structure like that instead of ultimate total size?
- deleted 7y ago[deleted]
- navaati 7y agoPlease do share what it was and why it happened, you can't leave us like that :) !
- ceejayoz 7y agoAWS is pretty flexible on first-time mistakes in a particular account; I've personal knowledge of several $5-10k "whoops" bills being nulled out. There may be a maximum threshold for this, though.
- karolisram 7y agoCan attest, once racked up $10k AWS bill due to a silly mistake, got it nulled. That was a great lesson about how fast things can go wrong with pay-as-you-go pricing if not monitored.
- lostlogin 7y agoCan you share some details?
- karolisram 7y agoIt's embarrassing to write that now but I accidentally left private key for EC2 instance publicly available on GitHub. And I think what happened is that a bot scraped that key and used my resources to mine Bitcoin.
- lostlogin 7y agoThis seems quite common. I have heard several stories to this effect. Faulty firewall settings or keys committed to the repo seem to be the common two.
- throwaway019254 7y agoI can confirm. I worked for AWS and we always waived all fees when people made mistake like this. Many thousands of dollars. We also asked them to summarize what happened so we can think how to help other users not to make the same mistake in future.
- CJefferson 7y ago
- deleted 7y ago[deleted]
- wolco 7y agoThat's the one aspect I liked. Never worrying about what the bill will look like. And with php it feels like it provides the best value / resources / cost.
- scarface74 7y agoMistakes happen. I’ve never heard a case where an honest mistake like that was made and with a simple email to AWS, they wouldn’t waive the charges.
- arcticfox 7y agoMight be true with AWS, but I had to go back and forth with GCP for days and threaten them with a blog post proving that their budget alert arrived 6 hours late (with our response in <15 minutes) in order to get refunded. Still pissed.
- scarface74 7y agoThere are reasons that Enterprises don’t trust GCP. Google doesn’t exactly have a sterling reputation for customer support. Anyone who stakes their business on any Google service should know what risks their taking. If their reputation isn’t enough to keep you from depending on GCP, their hiring of a lot of people from Oracle to beef up their sales staff should be a major warning.
- throwawaygcp329 7y agoNot only that, remember those mysterious "Google layoffs" back around Valentine's Day, before this virus mess? The ones they were so secretive about? That was one of the new Oracle execs firing the entire Seattle-based cloud marketing team. The reason for firing those people was solely to open up headcount to get more salespeople. The marketing team was not happy, for obvious reasons, and company PR worked pretty hard to spin this one. This is not how Googlers expect the company to work. But it is exactly what Oracle refugees expect. My take: GOOG is in for some Ballmer doldrum years of its own. They've well and truly arrived for the employees, but Wall Street hasn't quite figured it out yet.
- scarface74 7y agoAnd guess what the first thing to get cut is during a recession - advertising. Not to mention if VC funding dries up not only does it affect advertising budgets, it’s mostly startups that are crazy enough to go with GCP. Most major enterprises who are going to start a major migration would go to AWS or Azure.
- _bxg1 7y agoThey should give you the option to set a hard limit across your entire account, to prevent you from accidentally spending more money than you have. "If I try to spend more than $5k in a month, something has gone wrong, don't let me do that."
- umvi 7y agoSeems like circuit breakers should be a standard safety feature for automatically infinitely scaling computers. I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. However, searching around it seems like I can only get alerts when a $$$ threshold is passed, but AWS won't take any action to stop computing or anything. Please prove me wrong.
- samstave 7y agothey already are.... but you also need to put a bit more effort into it. For example, you shut down the ability to launch anything in a particular region easily - but assume you specifically want to exceed a default limit - you can call speak with your rep and have any of your limits set to whatever you want
- ghaff 7y ago>I would rather my whole system shut down and be unusable while I investigate vs. auto-scale and charge me a bill I can't cover. The counterargument is that you get a usage spike (which is often a good thing for a company), and AWS shuts down everything connected to your AWS account without warning. I'm not necessarily sure that optional/non-default hard circuit breakers would be a bad thing. But it certainly appears not to be a heavily demanded customer feature and, honestly, if it's not the default--which is shouldn't be--I wonder how many customers, or at least customers the cloud providers really care about, would use them.
- _bxg1 7y agoYou'd factor that in to the ceiling you set. Maybe your ceiling is 2x or 3x your expected usage. That could still be low enough not to bankrupt your company.
- 0x202020 7y agoI have been on the saving end of some of these AWS mistakes (billing/usage alerts are important people), but alerts aren't always enough to keep them from happening completely if they are fast - Not stopping AWS Glue with an insane amount of DPUs attached when not using it. Quote "I don't know I just attached as many as possible so it would go faster when I needed". - Bad queueing of jobs that got deployed to production before the end of month reports came out. Ticket quote "sometimes jobs freeze and stall, can kick up 2 retry instances then fails", not a big problem in the middle of the month when there was only a job once a week. End of the month comes along and 100+ users auto upload data to be processed, spinning up 100+ c5.12xlarge instances which should finish in ~2 mins but hang overnight and spin up 2 retry instances each - Bad queueing of data transfer (I am sensing a queueing problem) that led to high db usage so autoscaling r5.24xlarge (one big db for everything) to the limit of 40 instances
- tootie 7y agoNot that it killed us or anything, but we hired a Director of DevOps at my company who we tasked with the simple job of setting up a dev server for a Java REST server that would have like 6 concurrent users. It needed a cache, but no persistent database. A task beneath a director and one that the dev team would usually just do themselves, but he was here to show how to DevOps the right way and not be so ad hoc. He somehow managed to set this up to cost like $8000/mo after we have conservatively budgeted for $50. He was fired for myriad reasons and we spent like a week trying to figure out what he had done.
- K0SM0S 7y agoMay I ask what kind of background he had? Was it a hiring bet/mistake or was he fine on paper (and probably claiming way too much)?
- klenwell 7y agoFrom a favorite HN comment: When there is a lot of money involved, people self-select into your company who view their jobs as basically to extract as much money as possible. This is especially true at the higher rungs. VP of marketing? Nope, professional money extractor. VP of engineering? Nope, professional money extractor too. You might think -- don't hire them. You can't! It doesn't matter how good the founders are, these people have spent their entire lifetimes perfecting their veneer. At that level they're the best in the world at it. Doesn't matter how good the founders are, they'll self select some of these people who will slip past their psychology. You might think -- fire them. Not so easy! They're good at embedding themselves into the org, they're good at slipping past the founders's radars, and they're high up so half their job is recruiting. They'll have dozens of cronies running around your company within a month or two. https://news.ycombinator.com/item?id=18003253 https://news.ycombinator.com/item?id=18003253 I'm guessing something like the dynamic described here was involved. The silver lining here may be that he outed himself (literally) before he was able to build an empire of such incompetence.
- tootie 7y agoThat's not really it. Our company is small enough that I can talk one-on-one with the head of the tech department and I did give direct feedback about this person. That head of tech was responsible for the mishire, but also got rid of this person pretty quickly once all the feedback accumulated. My company is service-based and just over 1000 people. Timesheets equal billable hours. It's occasionally very pressurized and we lose people pretty quickly when there's a lull in work, but it also means that useless people have absolutely nowhere to hide.
- samstave 7y agoWTF? I had a new employee dev include AWS creds in his github which was pulled instantly by hacker bots that launched a SHITTON of instances globally and cost $100K in a matter of hours... It took my team like 6 hours to get it all under control... but AWS dropped all charges and we didnt have to pay any hosting costs for it.... So why didnt you work with AWS to kill such charges?
- winter_blue 7y agoHow does AWS collect on the $80k debt? If you're a startup, you could cancel/freeze your credit card, dissolve the LLC / S Corp, set a new one up with a similar name, and transfer all IP assets over. Poof - all debts and liabilities erased. What's wrong with this approach? It's not like they can collect on your personally, or go after the new company. (I wonder how they would even figure what legal entity is behind the new company/wesbite.)
- ceejayoz 7y agoYou'd run the risk of a court deciding to pierce your LLC's shield. https://www.nolo.com/legal-encyclopedia/personal-liability-piercing-corporate-veil-33006.html https://www.nolo.com/legal-encyclopedia/personal-liability-p...
- winter_blue 7y agoI read that, and the key thing to pay to attention is whether you willfully did something that was “unfair”, “unjust”, and “fraudulent”. I don’t think those apply here. If by sheer accident you were hit with a giant AWS bill, and you were facing potentially having to shut down your company, and you conducted the maneuver that I described, what’s wrong with it? Your company was facing a life-or-death situation, and decided to be reborn. Maybe there needs to be a form of corporate bankruptcy where the company can retain its core/key IP assets...
- nitrogen 7y agoIt's not like they can collect on your personally, or go after the new company. That is not a safe assumption to make, especially if you are deliberately (AKA fraudulently) dodging debts (IANAL).
- winter_blue 7y agoI’ve addressed this in this reply to a sibling comment stating that the LLC/corporate shield could be pierced here: https://news.ycombinator.com/item?id=22734033 https://news.ycombinator.com/item?id=22734033
- logicallee 7y agoFor startups that "might take off" I would use a dedicated virtual private server as a good alternative to unreasonable bills due to provisioning mistakes. You could be spending that time coding rather than than figuring out if you're making a mistake provisioning. There are too many stories like yours out there. Here is my referral code for the one I use: https://crm.vpscheap.net/aff.php?aff=15 https://crm.vpscheap.net/aff.php?aff=15 (I previously asked Dan, the mod here, if I can share in this way and he said it's okay. I don't have other affiliation with that company and have found it good.)
- freepor 7y agoAmazon doesn’t really want to kill startups using AWS even when they fuck up. If you had made some phone calls / worked LinkedIn / Tweeted / written a blog post you could have gotten that refunded in a week.
- Mirioron 7y agoBut you need to know this is possible in the first place. And once you've had to pay the price for it it's hard to ever feel safe using it again.
- ransom1538 7y ago? You should call AWS. They will negotiate. If it doesn't work out -- I think you should file a dispute with the credit card company. They will probably back you and want AWS to negotiate. If none of this happens - threaten to declare bankruptcy. Either way, AWS will back off. There is no money in it for them. IN NO WAY, should you pay the 80k bill and shut down your company.
- fulafel 7y agoTransferring money out from a company to hide from creditors is probably illegal.
- deleted 7y ago[deleted]
- perl4ever 7y agoWe read horror stories in the media because they make good stories, but in fact, people fuck up all the time in business and then it's just reversed because in most cases, nobody is irrationally out for their pound of flesh. People fat finger multimillion dollar trades on Wall Street and while I don't know that it's guaranteed to work out, I definitely have read about instances of that being reversed. If cryptocurrency and smart contracts make sense to you, you might not be aware that forgiveness for human error really does happen in normal business.
- manigandham 7y agoYes, you're legally obligated to pay the bill. You can't decide you don't want to pay because you don't like the cost. Claiming an unfounded dispute or transferring funds to a new company is fraud and you'll probably end up with both AWS and your bank coming after you for collections. With $80k on the line, its enough to file legal claims. The best plan is to negotiate directly with AWS and ask for forgiveness and a payment plan. Do not try to run away from your financial obligations or you will make it far worse. EDIT: you've rewritten your comment so this doesn't apply, but please don't recommend that people avoid their debts.
- zxcvbn4038 7y agoA lot of times your AWS rep can help with stuff like that. They are more interested in ongoing revenue then a one-time score that will end up going to collections. I had a situation where someone spun up an instance with an AMI that cost $$ per hour to run, then decided to go home early and disconnect for the weekend, so Monday morning I noticed we had been billed $22,000 for an idle instance running the AMI. It got handled. No worse then the kid who bought ten grand of “smurfberries” on his dad’s phone.
- paxys 7y agoIf shared PHP hosting was good enough for your use case why on earth were you on AWS (and running 4K video encoding clusters!) in the first place?
- tlarkworthy 7y agoThis is why I advise clients to setup billing alerts delivered to management (not tech team) as one of the first things todo when adopting a pay-as-you go technology [1] [1] https://www.futurice.com/blog/mastering%20bigquery%20costs https://www.futurice.com/blog/mastering%20bigquery%20costs
- msh 7y agoI dont understand why amazon does not provide hard billing limits, like reach x usd then shutdown everything, just for emergency.
- discordance 7y agobecause it's not in their business interests to do so?
- sireat 7y agoAre there any major cloud hosting/computing providers which do provide a hard spend limit? We've heard the horror stories from the Google data egress pricing "surprises" (like that GPT adventure game guy incurred a few month ago https://news.ycombinator.com/item?id=21739879 https://news.ycombinator.com/item?id=21739879). We've heard the AWS and Azure horror stories. It seems crazy that the only hope of correcting a mistaken overspend is a helpful support desk. The first one is free, right? At least AWS does have such a support desk, Azure may have and with GCP you are better off just shuting down the company. How about lesser providers such as Digital Ocean. Let's say your code mistakenly provisions 1000 droplets instead of 100. Is this a scenario you can prevent at an admin level?
- segfaultbuserr 7y agoScary. Does AWS allow one to set a hard price limit on your account today?
- partiallypro 7y agoI mostly have an Azure background but we absorbed a country that ran on AWS. I did an audit of all of their infrastructure this fall, and found they were over spending on AWS by 90%! I wish I had been tasked with it sooner, it could have saved the company hundreds of thousands, that's a few new hires, I was shocked at how mismanaged it was. It seems like the person that set it up was not familiar with cloud pricing models. For example, there were 100 or so detached disks that hadn't been touched in a year or two, which of course isn't free. The instances were too big, etc. I've always found Azure's billing to be easier to maintain cost controls over (at least it feels more friendly.) I wonder how many companies unintentionally over spend on cloud services because of the lack of understanding of the pricing models.
- _-___________-_ 7y agoYou absorbed a country?!
- 101404 7y agoThat's the reason I never went with cloud hosting. No kidding.