3 ms·
Every morning I would check AWS billing just out of habit. I'm just thankful I did - otherwise everything would have kept running... The lesson for me was don'
by appstorelottery 7y ago
Every morning I would check AWS billing just out of habit. I'm just thankful I did - otherwise everything would have kept running...
The lesson for me was don't trust your internally-hacked-together instance management system. The AWS interface to storage and instances is the base truth. And perhaps more importantly - I'm never getting into another startup which has financial risk like that without being a core expert in that risk/tech. I was focused on the business + client code - and had very little clue about the nitty-gritty of AWS. I should have been more involved with the code on that side, or at least the data-flow architecture.
- seibelj 7y agoAssuming you were incorporated and had a business account - declare bankruptcy and the bill goes away. I don’t understand why you would still pay the bill if you were going out of business anyway.
- appstorelottery 7y agoWhy didn't I file bankruptcy? This happened in Australia and declaring bankruptcy was not the right thing to do - for many reasons, not the least of which it makes it much harder to operate as a director of a previously bankrupt company, but in the worst case my bank would have just gone after me as I'd given a personal guarantee.
- garmaine 7y agoThere is no concept of limited liability in Australia?
- jkaplowitz 7y agoEven in the United States, most small business loans require personal guarantees which narrowly override the corporate limited liability to make that guarantor liable for that debt if the company doesn't pay. There are some rare exceptions, and possibly more for startups funded by big-name VCs, but I don't know.
- Scoundreller 7y agoBut this isn't a small business loan: it's a debt to Amazon.
- namdnay 7y agoExcept the loan money will go straight to Amazon, and you are now unable to repay the loan to the bank
- Scoundreller 7y agoDepends where Amazon ranks in seniority in bankruptcy (protection). You don't have to run out of money to file for it. Purdue Pharma sure didn't.
- vetinari 7y agoWhere exactly does the bank enter the picture? Scenario 1: Amazon will ask for the payment (if using cc); the bank will respond there are no funds in the account; Amazon deals directly with the company further directly, not with the bank, eventually getting payment order from the court. If the company went bankrupt meanwhile, Amazon might not get their money. Scenario 2: Amazon will send the invoice; invoice will not get paid. After due date, Amazon will contact the company directly; bank doesn't even enter the picture, until collection order comes from the court. If the company went bankrupt meanwhile, Amazon might not get their money. There's no scenario where some hypothetical loan would go straight to Amazon, unless Amazon has some instrument, that instruct the bank to pay them. Something like bank guarantee or promisory note, and uses them before declaring bankrupcy.
- Talanes 7y agoI think they were referring to a scenario where Amazon is draining the funds that have already been loaned. Thus Amazon already has their money, and the bank is the one coming after you during bankruptcy.
- 7y ago
- hnick 7y agoIf a director becomes personally bankrupt (such as trying to be the good guy and using personal guarantees to take on company debts in an effort to scrape through) then they're banned from running a company until it clears. If they're the director of a company that goes bankrupt, I believe they get 2 chances (companies) before there's a chance of being banned from running more for a time. Either way it might be nice to keep your options open, depending on your plans.
- scarface74 7y agoOr you could just send an email to support and ask them to waive the charges.
- lostlogin 7y agoIf that got to the right person on the right day and they knew it was going to kill the company, it seems likely to help. And combined with the fact that it would probably guarantee future revenue way off into the future...
- scarface74 7y agoI have never heard of a case where they wouldn’t give refunds. AWS is competing with the 95% of compute that is not running in the cloud (their own statistics). The last thing they want is a reputation that one mistake will bankrupt a business.
- Supermancho 7y ago> I have never heard of a case where they wouldn’t give refunds. Really? Working in Southern California a few years ago, refund requests were refused ALL THE TIME. This is why there's a common belief that what you are charged you simply owe them, period. It may be more progressive now, but let's not be revisionist.
- manigandham 7y agoWe had spot instances with a mistakenly high bid that incurred thousands overnight when the prices spiked. No refund offered. I know several other companies that had expensive mistakes without refunds. There's probably a complex decision tree for these issues and I doubt anyone really knows outside of AWS.
- staticassertion 7y agoI've repeatedly seen requests of this nature handled by AWS - 75% cuts to billing, 90% cuts even.
- 7y ago
- deleted 7y ago[deleted]
- justinclift 7y agoOn the other hand, it sounds like you hired someone who wasn't really up for the level of responsibility given. :( In theory ;), you shouldn't have to be a core expert in everything. But yeah... in the real world, things aren't so cut and dry. :/
- nitely 7y agoTBH, the real problem is AWS bills cannot be capped in any way (you can setup an alarm, though). It's unreasonable to expect a programmer won't make mistakes.
- manigandham 7y agoOf course they can be capped, you just turn off the services. If you're asking them to automate that for you, then the counterpoint would be people accidentally setting a budget that wipes out their resources and complaining about that. Easier for both sides to just ask AWS for a refund if there's a reasonable case.
- nicoburns 7y ago> the counterpoint would be people accidentally setting a budget that wipes out their resources and complaining about that. This wouldn't be an issue if it was configurable.
- manigandham 7y agoMistakes will always be an issue. How you recover is more important. Would you rather make a mistake leading to a big bill with the possibility of a refund or set your max budget and have your resources permanently deleted?
- nicoburns 7y agoThere would be no need to delete existing resources. Just prevent me from creating new ones until action is taken. For small projects in particular, I'd much rather have service taken offline and an email notification than even a $1000 bill. And $1000 is small in the scale of what you could end up with on AWS.
- lowercased 7y ago> I'm never getting into another startup which has financial risk like that without being a core expert in that risk/tech This may be something that is 'unstated', but unless you actually had access to fix something that was wrong, as well, being an expert in that wouldn't really help all that much. I've been in situations where I have explicit/expert knowledge of XYZ, but when the people responsible for XYZ do not take your input, and/or don't provide you the ability to fix a problem, expert knowledge is useless (or worse, it's like having to watch a train wreck happen when you know you could have stopped it).
- Aeolun 7y agoThis. But on the other hand, you can be ready with the popcorn when shit eventually does hit the fan.
- glenngillen 7y agoDid you reach out to AWS support or your account manager? They’d definitely have worked something out.
- epiphanitus 7y agoSRE here. I feel for your situation. Here's some advice. One simple thing you could do is set up AWS billing alarms and have them delivered to a notification app like PagerDuty. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori.. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori.... If you don't want to pay for PD, you can patch together any number of ways to get your phone to scream and holler when it gets an email from ohshit@amazonasws.com. It's also good to have clear expectations as to whose responsibility it is to deal with problem x between the hours of y and z and exactly what they are supposed to do. Keep the alerts restricted to the really important stuff, because if your team becomes overloaded with useless alerts they will 1) dislike you and 2) be more prone to accidentally mistaking a five alarm fire for a burnt casserole. There are more complex systems you could build, but that's a start.
- HenryBemis 7y agoThank you for this. How can anyone run ANY service with ANY company and not add a clause in the contract (and then have the alerts up an running) in controlling costs? I remember PagerDuty was advertising (a lot) on Leo Laporte's podcasts a few years back. A clause in the contract: if monthly bill reaches $Xk amount then: (a) seek written approval by client, and (b) continue until $Yk or approval is given with a new ceiling price.
- MattSayar 7y agoI was just playing around with AWS a while ago and was surprised that I could not find any option to put a cap on the amount I'd spend in a month. Only thing I could do was set up alerts. I imagine AWS would have 0 problems suspending all my services if I can't pay, so why can't it do the same thing when it reaches my arbitrary cap?
- 101404 7y agoAWS should have a cost cap. Set a max spend value and shut down all servers if you spent it.
- dragonwriter 7y ago> AWS should have a cost cap. Set a max spend value and shut down all servers if you spent it. That might make sense for some particular services (e.g., capping the cost on active EC2 instances) but lots of AWS costs of data storage costs, and you probably don't want all your data deleted because you ran too many EC2 instances and hit your budget cap. Where exactly you are willing to shut off to avoid excess spend and what you don't want to sacrifice automatically varies from customer to customer, so there's no good one-size-fits-all automated solution.
- JamesBarney 7y agoI think if resources had an option of "At cap: Do nothing, Shut down, shutdown and erase data" that would cover most of the use cases.
- jfkebwjsbx 7y agoKeeping the data for a week but completely inaccessible would not be a huge cost for AWS yet a big relief for startups.
- deleted 7y ago[deleted]
- shawabawa3 7y agoDid you contact AWS and let them know it was a mistake? They have a good track record of cancelling huge bills the first time they happen
- samstave 7y agoWe used to have a bunch of billing graphs in stack driver with alerting thresholds to pagerduty to capture exactly situations like this.