12 ms·
What was the mistake?
by jacobra2 7y ago
What was the mistake?
- appstorelottery 7y agoRunning up a shitload of instances for testing and leaving all of them running overnight. Each of these instances continually rendered 4k video data to storage. This kind of test was supposed to be 1000x smaller, running for at most 10-20 seconds at time. He had written his own provisioning system which - according to his report - failed to properly manage instances "weird" edge case. No kidding.
- appstorelottery 7y agoEvery morning I would check AWS billing just out of habit. I'm just thankful I did - otherwise everything would have kept running... The lesson for me was don't trust your internally-hacked-together instance management system. The AWS interface to storage and instances is the base truth. And perhaps more importantly - I'm never getting into another startup which has financial risk like that without being a core expert in that risk/tech. I was focused on the business + client code - and had very little clue about the nitty-gritty of AWS. I should have been more involved with the code on that side, or at least the data-flow architecture.
- seibelj 7y agoAssuming you were incorporated and had a business account - declare bankruptcy and the bill goes away. I don’t understand why you would still pay the bill if you were going out of business anyway.
- appstorelottery 7y agoWhy didn't I file bankruptcy? This happened in Australia and declaring bankruptcy was not the right thing to do - for many reasons, not the least of which it makes it much harder to operate as a director of a previously bankrupt company, but in the worst case my bank would have just gone after me as I'd given a personal guarantee.
- garmaine 7y agoThere is no concept of limited liability in Australia?
- jkaplowitz 7y agoEven in the United States, most small business loans require personal guarantees which narrowly override the corporate limited liability to make that guarantor liable for that debt if the company doesn't pay. There are some rare exceptions, and possibly more for startups funded by big-name VCs, but I don't know.
- Scoundreller 7y agoBut this isn't a small business loan: it's a debt to Amazon.
- namdnay 7y agoExcept the loan money will go straight to Amazon, and you are now unable to repay the loan to the bank
- Scoundreller 7y agoDepends where Amazon ranks in seniority in bankruptcy (protection). You don't have to run out of money to file for it. Purdue Pharma sure didn't.
- vetinari 7y agoWhere exactly does the bank enter the picture? Scenario 1: Amazon will ask for the payment (if using cc); the bank will respond there are no funds in the account; Amazon deals directly with the company further directly, not with the bank, eventually getting payment order from the court. If the company went bankrupt meanwhile, Amazon might not get their money. Scenario 2: Amazon will send the invoice; invoice will not get paid. After due date, Amazon will contact the company directly; bank doesn't even enter the picture, until collection order comes from the court. If the company went bankrupt meanwhile, Amazon might not get their money. There's no scenario where some hypothetical loan would go straight to Amazon, unless Amazon has some instrument, that instruct the bank to pay them. Something like bank guarantee or promisory note, and uses them before declaring bankrupcy.
- scarface74 7y agoOr you could just send an email to support and ask them to waive the charges.
- lostlogin 7y agoIf that got to the right person on the right day and they knew it was going to kill the company, it seems likely to help. And combined with the fact that it would probably guarantee future revenue way off into the future...
- scarface74 7y agoI have never heard of a case where they wouldn’t give refunds. AWS is competing with the 95% of compute that is not running in the cloud (their own statistics). The last thing they want is a reputation that one mistake will bankrupt a business.
- Supermancho 7y ago> I have never heard of a case where they wouldn’t give refunds. Really? Working in Southern California a few years ago, refund requests were refused ALL THE TIME. This is why there's a common belief that what you are charged you simply owe them, period. It may be more progressive now, but let's not be revisionist.
- manigandham 7y agoWe had spot instances with a mistakenly high bid that incurred thousands overnight when the prices spiked. No refund offered. I know several other companies that had expensive mistakes without refunds. There's probably a complex decision tree for these issues and I doubt anyone really knows outside of AWS.
- staticassertion 7y agoI've repeatedly seen requests of this nature handled by AWS - 75% cuts to billing, 90% cuts even.
- 7y ago
- deleted 7y ago[deleted]
- justinclift 7y agoOn the other hand, it sounds like you hired someone who wasn't really up for the level of responsibility given. :( In theory ;), you shouldn't have to be a core expert in everything. But yeah... in the real world, things aren't so cut and dry. :/
- nitely 7y agoTBH, the real problem is AWS bills cannot be capped in any way (you can setup an alarm, though). It's unreasonable to expect a programmer won't make mistakes.
- manigandham 7y agoOf course they can be capped, you just turn off the services. If you're asking them to automate that for you, then the counterpoint would be people accidentally setting a budget that wipes out their resources and complaining about that. Easier for both sides to just ask AWS for a refund if there's a reasonable case.
- nicoburns 7y ago> the counterpoint would be people accidentally setting a budget that wipes out their resources and complaining about that. This wouldn't be an issue if it was configurable.
- manigandham 7y agoMistakes will always be an issue. How you recover is more important. Would you rather make a mistake leading to a big bill with the possibility of a refund or set your max budget and have your resources permanently deleted?
- nicoburns 7y agoThere would be no need to delete existing resources. Just prevent me from creating new ones until action is taken. For small projects in particular, I'd much rather have service taken offline and an email notification than even a $1000 bill. And $1000 is small in the scale of what you could end up with on AWS.
- lowercased 7y ago> I'm never getting into another startup which has financial risk like that without being a core expert in that risk/tech This may be something that is 'unstated', but unless you actually had access to fix something that was wrong, as well, being an expert in that wouldn't really help all that much. I've been in situations where I have explicit/expert knowledge of XYZ, but when the people responsible for XYZ do not take your input, and/or don't provide you the ability to fix a problem, expert knowledge is useless (or worse, it's like having to watch a train wreck happen when you know you could have stopped it).
- Aeolun 7y agoThis. But on the other hand, you can be ready with the popcorn when shit eventually does hit the fan.
- glenngillen 7y agoDid you reach out to AWS support or your account manager? They’d definitely have worked something out.
- epiphanitus 7y agoSRE here. I feel for your situation. Here's some advice. One simple thing you could do is set up AWS billing alarms and have them delivered to a notification app like PagerDuty. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori.. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori.... If you don't want to pay for PD, you can patch together any number of ways to get your phone to scream and holler when it gets an email from ohshit@amazonasws.com. It's also good to have clear expectations as to whose responsibility it is to deal with problem x between the hours of y and z and exactly what they are supposed to do. Keep the alerts restricted to the really important stuff, because if your team becomes overloaded with useless alerts they will 1) dislike you and 2) be more prone to accidentally mistaking a five alarm fire for a burnt casserole. There are more complex systems you could build, but that's a start.
- HenryBemis 7y agoThank you for this. How can anyone run ANY service with ANY company and not add a clause in the contract (and then have the alerts up an running) in controlling costs? I remember PagerDuty was advertising (a lot) on Leo Laporte's podcasts a few years back. A clause in the contract: if monthly bill reaches $Xk amount then: (a) seek written approval by client, and (b) continue until $Yk or approval is given with a new ceiling price.
- MattSayar 7y agoI was just playing around with AWS a while ago and was surprised that I could not find any option to put a cap on the amount I'd spend in a month. Only thing I could do was set up alerts. I imagine AWS would have 0 problems suspending all my services if I can't pay, so why can't it do the same thing when it reaches my arbitrary cap?
- 101404 7y agoAWS should have a cost cap. Set a max spend value and shut down all servers if you spent it.
- dragonwriter 7y ago> AWS should have a cost cap. Set a max spend value and shut down all servers if you spent it. That might make sense for some particular services (e.g., capping the cost on active EC2 instances) but lots of AWS costs of data storage costs, and you probably don't want all your data deleted because you ran too many EC2 instances and hit your budget cap. Where exactly you are willing to shut off to avoid excess spend and what you don't want to sacrifice automatically varies from customer to customer, so there's no good one-size-fits-all automated solution.
- JamesBarney 7y agoI think if resources had an option of "At cap: Do nothing, Shut down, shutdown and erase data" that would cover most of the use cases.
- jfkebwjsbx 7y agoKeeping the data for a week but completely inaccessible would not be a huge cost for AWS yet a big relief for startups.
- deleted 7y ago[deleted]
- shawabawa3 7y agoDid you contact AWS and let them know it was a mistake? They have a good track record of cancelling huge bills the first time they happen
- samstave 7y agoWe used to have a bunch of billing graphs in stack driver with alerting thresholds to pagerduty to capture exactly situations like this.
- yingw787 7y agoHoly crap dude, that's some nightmare shit right there. Does AWS update the billing console per day or upon request? I get charged per month, but I should add a habit in my habit tracker to learn more about my expenses...
- freeone3000 7y agoHourly. You can also set up billing alerts, which will email you.
- lostlogin 7y agoThis is what was needed.
- WrtCdEvrydy 7y agoBe aware that some services bill asynchronously so it can take 24 hours in some instances.
- user5994461 7y agoHow many is a shitload of instances? Are we talking tens, hundreds, thousands? In my experience AWS had very stringent limits on the amount of active instances of each type (starts around 10 for new accounts, 2 for the more expensive instances). It takes tickets to support then days of waiting to raise these limits. That should have prevented your company from creating tens of instances, let alone hundreds, unless that's already your typical daily usage.
- redis_mlc 7y agoThere used to be no limit on EC2 instances.
- deleted 7y ago[deleted]
- x86_64Ubuntu 7y agoWhat's the technical process to ensure that this never happens? Nowadays, having to have someone "watch" the test and then kill the instances is manual labor which is a no-no. So how do you make it so that your test fires up the instances, and then kills them when the test is done.
- dev_throw 7y agoYou can use auto scaling groups with a load balancer to terminate instances when not in use and spin them up as required.
- rcxdude 7y agoYou can do timed instances, and/or make the instances have timed job to shutdown after a fixed time (which is what I use to shut down an instance which only gets spooled up for occasional CI jobs after an hour).
- bluecmd 7y ago+1. When I had to use AWS for batch workloads, which at the time at least didn't have a TTL attribute on VMs, I made sure that the VM first scheduled a shutdown in like 30 min if the test was supposed to only run in 10 min.
- Twirrim 7y agoNot sure if this would help in this particular scenario, but unit and integration testing of operations scripts can save a lot of pain, anguish and $$s too. It's horrifying how many places treat writing tests for services as critical, but then completely fail to write tests for their operational tooling. Including tools responsible for scaling up and down infrastructure, deleting objects etc.
- dirtydroog 7y agoBut if a test fails does it now mean you're bankrupt?
- tomerico 7y agoWhy is there no way to set a limit on billing on AWS? Especially for cases like this, where killing testing instances does not have a dramatic negative effect...
- strongbond 7y agoAgreed. The simple solution is an expenditure cap. Why can't Amazon implement one? The fear of it going wrong like this would make me keep away from AWS forever.
- joshvm 7y agoNowadays quotas give you some safety net. For example you usually have to request more than one GPU to avoid burning money that way, or more than say 32 instances. It should not be possible for a new account to spawn 1k VMs overnight. The problem with billing is that often these charges are not calculated instantly, and others are not trivial to deal with. For example what happens if you go over budget on bandwidth or bucket storage, but still within quota? What do you kill? Do you immediately shut down everything? Do you lose data? There are lots of edge cases. You can normally write your own hooks to monitor billing alerts and take action appropriately.
- ehsankia 7y agoWait, is there really not one on AWS? I thought this was the #1 most important feature on any such cloud systems. It's the very very first thing I set when setting up my GCloud hobby project. I was like, this is fun and all, but I don't care about this enough so I limited it to 3$ per day and 50$ per month. If it goes above, I'm very happy to let it die, and it also gives me a warning so I know something is up. The 2 times it triggered, there was something I managed to fix so the tool is still up and running costing pennies.
- WrtCdEvrydy 7y agoThis is why it's Terraform or nothing for me.
- scubbo 7y agoI'd be fascinated to hear how Terraform would have intelligently known that those instances were not meant to stay on overnight.
- WrtCdEvrydy 7y agoHonestly, I'd create the instances using an ASG, then set the ASG size to 0 (or throw inside a while loop until any errors go away). Always create instances from an AMI and always put them in an ASG (even if the ASG only has 1 item min, target, and max on it).
- jenkinstrigger 7y agoI love Terraform and ASGs but that still doesn't solve the fact that their SRE overprovisioned. They might have even used both things!
- deleted 7y ago[deleted]
- smiths1999 7y agoThis has happened to me several times, albeit at a much smaller scale. I fire up a few GPU instances for training neural networks and when I got to shut the instances down I forget that you always need to refresh the instance page before telling AWS to stop my instances. I still go through all the confirmations saying I do, indeed, want to stop all instances. However, these few times I forgot to refresh to make sure they actually were shutting down and simply went to bed. Not an $80k mistake, but certainly a couple hundred dollars, which hurts as a grad student. Now I have learned, _always_ refresh the page and instance list prior to shutting anything down and _always_ confirm the shutdown was successful.
- quickthrower2 7y agoUsing a post paid service and getting bill shock.
- BluePen7 7y agoNot who you asked, but my mistake was transferring an S3 bucket full of unused old customer web assets to glacier, we were paying a lot to host them each month, and weren't using them anymore. I set the lifecycle rule on all objects in the bucket, for as soon as possible (24 hours). About 2 days later first thing in the morning I get a bunch of frantic messages from my manager that whatever script I was running, please stop it, before I'd even done anything for the day. The lifecycle rule had taken effect near the end of the previous day, and he was just getting all the billing alerts from overnight, it was all done. I read about glacier pricing, but didn't realize there was a lifecycle transfer fee per 1000 objects (I forget the exact price, maybe $0.05 per 1000 objects). That section was a lot further down the pricing page. The bucket contained over 700 million small files. I'd just blown $42,000. That was over a month's AWS budget for us, in the end AWS gave us 10% back. On the plus side, I didn't get in too much trouble, and given we'd break even in 4 years on S3 costs, upper management was gracious enough to see it as an unplanned investment. TLDR: My company spent 42k for me to learn to read to the bottom of every AWS pricing page.
- Nition 7y agoWhat would have been the correct solution here? Group them into compressed archives first to reduce file count?
- jenkinstrigger 7y agoOne .zip to rule them all :)
- Nition 7y agoHaha, I original wrote "one giant zip file?" but I decided to rephrase it as a more serious answer.
- stainforth 7y agoWhy would they create a pricing structure like that instead of ultimate total size?