4 ms·
The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and
by acoustics 1y ago
The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent.
Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build.
I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend?
Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?
- rr808 1y agoMaybe, but its a huge reason to use real servers instead of serverless.
- pixl97 1y agoI mean real servers get hit with things like bandwidth fees so it's not a 100% solution.
- estimator7292 1y agoNot even remotely the same scale of problem. Like at all. If your business suddenly starts generating Tbs of traffic (that is not a ddos), you'd be thrilled to pay overage fees because your business just took off. You don't usually get $10k bandwidth fees because your misconfigured service consumes too much CPU. And besides that, for most of these cases, a small business can host on-prem with zero bandwidth fees of any type, ever. If you can get by with a gigabit uplink, you have nothing to worry about. And if you're at the scale where AWS overages are a real problem, you almost certainly don't need more than you can get with a surplus server and a regular business grade fiber link. This is very much not an all-or-nothing situation. There is a vast segment of industry that absolutely does not need anything more than a server in a closet wired to the internet connection your office already has. My last job paid $100/mo for an AWS instance to host a GitLab server for a team of 20. We could have gotten by with a junk laptop shoved in a corner and got the exact same performance and experience. It once borked itself after an update and railed the CPU for a week, which cost us a bunch of money. Would never have been an issue on-prem. Even if we got DDoSed or somehow stuck saturating the uplink, our added cost would be zero. Hell, the building was even solar powered, so we wouldn't have even paid for the extra 40W of power or the air conditioning.
- Nextgrid 1y agoDepends where you order your server. If you order from the same scammers that sell you "serverless" then sure. If you order from a more legitimate operator (such as literally any hosting company out there) you get unmetered bandwidth with at worst a nasty email and a request to lower your usage after hitting hundreds of TBs transferred.
- herpdyderp 1y agoThe solution is simple: budget caps.
- mlhpdx 1y agoIs it simple? So what happens when you hit the cap, does AWS delete the resources that are incurring the cost and destroy your app? Imagine the horror stories on Hacker News that would generate.
- ndsipa_pomu 1y agoSurely that's the fault of the purchaser setting the cap too low. Maybe rather than completely stopping the service, it'd be better to rate limit the service when approaching/reaching the cap.
- mlhpdx 1y agoUsing that logic, isn’t it the fault of the user to set up an app without rate limiting?
- ndsipa_pomu 1y agoIt's misleading to promote a free tier that can then incur huge charges without being able to specify a charge cap.
- ninalanyon 1y agoIf it can incur any charges at all then it isn't free.
- zorked 1y agoStop accepting requests like has been the case since the beginning of time?
- clvx 1y agoYes, that’s exactly the expected behavior. It can alert if it’s closed to threshold. Very straightforward from my point of view.
- cm2187 1y agoHow about spending caps / circuit breakers? Doesn't seem an unsolveable problem to me.
- nostrebored 1y agoThen you’re the person who took down their small business when they were doing well. At AWS I’d consistently have customers who’d architected horrendously who wanted us to cover their 7/8 figure “losses” when something worked entirely as advertised. Small businesses often don’t know what they want, other than not being responsible for their mistakes.
- Touche 1y agoEveryone who makes this argument always assumes that every website on the internet is a for-profit business when in reality the vast majority of websites are not trying to make any profit at all, they are not businesses. In those cases yes absolutely they want them to be brought down.
- jiggawatts 1y agoOr instead of an outage, simply have a bandwidth cap or request rate cap, same as in the good old days when we had a wire coming out of the back of the server with a fixed maximum bandwidth and predictable pricing.
- acoustics 1y agoThere are plenty of options on the market with fixed bandwidth and predictable pricing. But for various reasons, these businesses prefer the highly scalable cloud services. They signed up for this
- wahnfrieden 1y agoEvery business has a bill they are unprepared to pay without evaluating and approving budget, even under successful conditions and even if that approval step is a 10 second process. It's obvious that Amazon does not add this because of substantial profit over any other concern.