10 ms·
The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a
by onion2k 6y ago
The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.
- jsiepkes 6y agoNot only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.
- serial_dev 6y agoIt is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be different.
- greatgib 6y agoIt's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.
- onion2k 6y agoThey want to suck the maximum money from consumers before they realize. I have very little money so I just don't use their services because a mistake would be disastrous. They might be losing out on me making a unicorn app on their platform. It's unlikely, but while the possibility of catastrophe exists I'll stick to not using them. That extends to not recommending anyone uses them either in case the worst happens.
- mikestew 6y agoI have very little money... Then the harsh reality is: companies don't care. Yeah, your app might turn out to be a unicorn, but the overwhelming odds are that it won't. And no one cares that you'll tell your other broke friends to avoid the service. We'd all like to think it to be different, that a company might care about appeasing my broke ass. But as already pointed out, they want the whales. I also wonder, despite the number of years "cloud services" have been around, if companies aren't still trying to figure out a gazillion other things and limiting customer spend might be a bit low on the priority list.
- thanksforthe42 6y agoMeanwhile this leaves an opportunity for a different company to provide these services. I do my best to avoid FAANG giants who don't think about me.
- renewiltord 6y agoThe highly price sensitive customer will force you to compete only on price. That's just forcing yourself into a commodity market. It's bad business. I would never try to cater to that market. Very dangerous. Competition will drive margins down to near zero.
- uoaei 6y ago> It is baffling why cloud providers don't have that option. ...is it? If a lazy dev leaves their corporate account open and you can bill it for their negligence, protected by the contract you already signed, you earn a lot of money. From a purely business perspective, it is stupid(!) to provide a stopgap for that. Edit: to be clear I am not advocating one way or the other. But it is surprising that people are "baffled" by this obvious profit optimization.
- camjohnson26 6y agoGoogle is around a trillion dollar company, your $75,000 is a completely immaterial amount to them. Not to mention it would be a one time payment that would drive away customers and lead to bad PR like this post.
- AntiImperialist 6y agoExcept Google has 4 million customers. If 1% of their customers made a 75k mistake, they make 3 billion dollars.
- uoaei 6y agoSo... where else are you going to go if you don't like these policies? If everyone has this policy, Google, Amazon, Microsoft, and the rest are in a good place. And suddenly it's the "industry standard." This hypothetical is already enacted today...
- brundolf 6y agoFor hobby projects you probably don't need auto-scaling, and should use a provider that charges a fixed monthly rate. You'll "waste" a little bit of money on unused uptime, but for a hobby project it will be a minuscule amount.
- vp8989 6y agoBut then I don't get to be a "serverless hero" and write blog posts about how my side project (that no one uses) costs $0.000034 to run instead of $5.
- gonzo41 6y agoI wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars. If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a raspberry pi on your desk. But really the first lesson you should learn in any cloud setup is Billing Alarms :) If you're doing ML or CV work then it's probably cheaper to build on the desktop and port to cloud once you understand what the workloads are.
- onion2k 6y agoFor AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake is costing $1000/minute and you're an hour away from a computer you have a very expensive problem.
- gonzo41 6y agoYou can trigger events from alarms. And Lambda's only last 15 minutes. So still cheaper than 75K :D.
- welkie 6y agoThat's not a bad idea. You could set it up to delete all Lambdas (assuming you've got a CI/CD system capable of redeploying them quickly later) if the billing goes over. Of course, this may hurt you more because of the outage it would cause. Up to you really.
- jjk166 6y agoSo you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?
- ZephyrBlu 6y agoProbably because it's not so simple on the backend. I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update. Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time. It would be a great feature, but I can imagine it being very complex. It's also probably cheaper for them to just wave away excess usage like this instead of building out a solution.
- donmcronald 6y agoAzure has it for some plans [1], but not others like pay-as-you-go. It seems arbitrary. 1. https://azure.microsoft.com/en-us/support/legal/offer-details/ https://azure.microsoft.com/en-us/support/legal/offer-detail...
- kevsim 6y agoI think that's not really an issue though is it? If you say "never charge me more than $100" they can a) ensure they never charge you more than $100 and b) work to optimize their own systems so that they cut you off as close to $100 as humanly possible. In the beginning they might eat some costs since it takes them a day to catch it, but they could work over time to bring that down. And it's not like it's costing GCP/AWS/Azure "sticker price" to provide their services.
- a-priori 6y agoThis is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount. If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get a bill for $1000. With an actual hard limit you create a financial incentive for the provider to minimize this overrun. Yes it might be difficult to fix but I assure you, if hard limits existed, the technical issues would be solved soon enough because now there's a reason to invest in a solution.
- 6y ago
- donmcronald 6y agoThis is my worst nightmare. Lol. I guess now is a great time to give Azure a shoutout for sitting on their hands for 8 years without so much as a response to the community for half a decade [1]. At least AWS allows using a prepaid credit card so they’ll need to call me if things go haywire. I bet if that $72k charge went through it would have been much harder to get out of. “Sorry, we don’t have the money” is a much better negotiating position than “can we please have our money back?” 1. https://feedback.azure.com/forums/170030-signup-and-billing/suggestions/3238796-spending-limit-or-maximum-cost-cap-for-azure https://feedback.azure.com/forums/170030-signup-and-billing/...
- dvfjsdhgfv 6y ago> “Sorry, we don’t have the money” is a much better negotiating position than “can we please have our money back?” I agree but why would you like to be in either position anyway? The so-called cloud services are terribly overpriced when compared to traditional servers.
- treeman79 6y agoDone correctly they save a lot of IT time. Seem companies hire five 6 figure people to try and cut amazon bill by a couple of grand a month. Never understood spending 50-100k a month to maybe save 5k
- Retric 6y agoIt’s often a fixed vs ongoing cost question. Spending 200k to save 5k per month breaks even in 3.4 years. However, for growing companies that 5k/month AWS premium can hit 200+k/month very quickly
- foolmeonce 6y ago>Done correctly they save a lot of IT time Not really, computing done correctly is about avoiding all of the pitfalls and finding ways to get zero cost benefits, free computation out of necessary redundancy, etc. Selling cloud computing is about creating options around every pitfall and finding ways to charge for every mitigation that will be necessary and charge for redundancy in the mitigation strategy for the mitigation strategy.. Even if you pay for all the redundant managed blah they offer to not lose your business by having any single point of technical failure in their network, their billing and IAM are your single points of failure, if you diversify to multiple clouds all the guarantees either cloud offers is now pointless redundancy so you are paying 10X pricing for an inadequate redundancy layer. If you look at Google's own model for computing, they didn't fall for this themselves, the computers they used were intentionally unreliable to not recursively pay for reliability and redundancy at any layer that can't provide the needed guarantee. You can basically go all in with one of these clouds and become a franchise add-on with roughly the same rights as your average mcdonald's store owner, or you are managing a strategy that is far more complex because of the complexity of these offerings than just using metal and free software.
- nojito 6y agoPrice transparency is the antithesis to the "cloud" and it's current financial success.
- jasonpeacock 6y agoIt's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive resources usually don't have massive cost spikes while active resources do, so you can better "protect" your passive resources by setting more aggressive cost limits on the active resources. This quickly gets more complicated. Another example is most monitoring services are a combination of active (actual metric monitoring) and passive (metric history) resources. A cost limit on that monitoring service likely won't provide sub-service granularity, mostly depending on whether the service even has different charges for monitoring vs history. Oh, also, even for a passive resource like storage, you also have active resource charges whenever you upload/download your data. Ugh, what a mess. The best thing to do is pay attention to your spending, just like you do with your personal & corporate budget.
- Closi 6y agoIt's almost like you could make it configurable so users can choose what happens if they go over, and to what extent.
- AngusH 6y agoBut we've had disk quotas before that mostly worked? If anything it seems an easier problem than processor time. I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that. Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails. I don't think they deleted files, it was just you couldn't write more than [quota] bytes to your disk. Is there something particular about cloud based systems that prevent this from working? ie. is this a specific problem with distributed storage? edit:tone
- robrtsql 6y agoS3 costs money to keep your files in, even if you're not touching them, so just preventing further uploads wouldn't do much to prevent your AWS bill from increasing.
- sandGorgon 6y agoGoogle does have this feature https://cloud.google.com/billing/docs/how-to/budgets-programmatic-notifications https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
- lights0123 6y ago> There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget. This looks like it has the same problems as the post, because it also relies on those budget alerts that can happen a long while after you've exceeded them.
- mgkimsal 6y agoVery late to the post, but this seems like "eventually consistent billing". distributed systems seem to rely on "eventual consistency" but... "eventual consistency" is not what most people want in billing threshold scenarios...
- modeless 6y ago"Following the steps in this capping example is not a guarantee that you will not spend more than your budget." "Resources [...] might be irretrievably deleted." Also it's not automatic, you have to manually write code to do it, and test it, and make sure not to break it. A reasonable implementation of this feature would be built into the console, guarantee a maximum spend, not require writing your own fallible code, and provide an option to preserve storage (at normal cost) so that all your data isn't deleted when your compute/API stuff is shut down.
- deleted 6y ago[deleted]
- 6y ago
- bpodgursky 6y agoThere are some cloud services where it's not quite this simple. S3 -- you can't just delete customer data because they hit a billing limit RDS -- not going to drop databases on the 27th of the month Anything with persistent data is going to have to stay alive and accumulate costs. Admittedly these services aren't where the crazy bills come from, but it does make a simple kill switch a bit more complex.
- raphaelj 6y agoYou don't have to immediately delete customer data. Most service that has a limit cap will have a "grace period" of a couple of days during which the service does not work but the data is not deleted. That give your some time to get notified of the issue, and fix the problem/increase the limit.
- heavyset_go 6y agoThis is a solved problem for every other service out there. You don't just delete the data, you give the customer a few days, weeks, or a month to pay their bill and if they don't, then you delete their data.
- betwixthewires 6y agoThe problem with this though is it opens a vector for exploitation: users could just use the grace period to store data for free for a period of time. This can quickly become a heavy financial burden if enough people do it. You could factor that into the price, but then you're potentially making the price point even more unattractive to users than it already is, and users that are responsible with their budgets would be subsidizing those that aren't. Not a very workable solution. I'd say a good solution is giving customers the option to stop accruing more storage capacity, and to have a max deadline accounted for in their budget to store data (basically each customer decides whether or not to pay for a grace period).
- heavyset_go 6y agoI've accidentally let my OVH subscription go unpaid, and they gave a 7 day window to pay my invoice or delete my data. That's seems pretty fair to me, and they seem to have wide enough margins to eat the cost and still offer some of the cheapest prices out there right now.
- trymas 6y agoAFAIK digitalocean has notification if you go over user defined limit.
- raphaelj 6y agoIt's ever worse for services like AWS Cloudfront. One of your competitors could just rent a cheap server on OVH with uncapped transfer and incur you $10k in cost in a few hours.
- cambalache 6y agoMaybe that it is your cue to move your server from AWS to OVH* * I dont have any idea about OVH
- tcoff91 6y agoCloudFront is a CDN. What the poster you've replied to is talking about is a competitor setting up a server that repeatedly downloads your content to rack up a huge CDN bill. OVH is not a suitable replacement for a CDN so you can't migrate from Cloudfront to an OVH server because a server is not a viable replacement for a CDN.
- cambalache 6y agoThat should be illegal, but hey, at least they support noble causes, so let them be. It sounds cynic but this is their game.
- betwixthewires 6y agoTo me this is akin to the personal checking overdraft scams banks were running for many years until those practices were made illegal. There is no engineering hurdle that is a valid excuse for allowing a customer to go over their stated budget by 86 million percent.
- jyu 6y agoThere is an easy explanation: It's hard to build this feature, there is no pressing demand from upper management, it's easier to get promoted doing other simpler projects. Think about what a real time snapshot means: you need to know how much of all the services are being used, project that in the future and compute the costs.
- betwixthewires 6y agoReally, it is a bit disappointing to see a bunch of engineers in this thread talking like this is some monumental, borderline unsolveavle problem. The solution is pretty easy to figure out, even taking into consideration different needs of different customers. The implementation might not be trivial, and legal liability questions might have to be considered beforehand, but the problem is not that hard.
- rk06 6y agoFWIW, Azure has that option