33 ms·

Accident Forgiveness

310 points by piperswe 2y ago

lagniappe 2y ago
This shouldn't be a selling point, it should just be table stakes.
- sealeck 2y ago
  You can say this for a whole bunch of new features people role out but this doesn't erase the fact that they are very much not the stakes and introducing them is great because it raises them!
- tptacek 2y ago
  We agree.
morningsam 2y ago
>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs. Good on Fly.io for trying to handle such situations more sensibly.
- bdcravens 2y ago
  There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
  - paulgb 2y ago
    I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
    - bdcravens 2y ago
      It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.
      - skeeter2020 2y ago
        this makes sense; plug the hole and stop taking on water before you ask for a free cleanup.
  - perchlorate 2y ago
    You likely already know that, but to anyone else interested: a good way to prevent these kinds of situations is to run 'nosey parker' on your git repo before pushing it to a remote. It will dig through your code and configs, looking at files and through all the git history, and highlight anything that looks like tokens, passwords, keys, etc. You can set it as a pre-commit hook to block the offending code from even being committed. https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
    - Aeolun 2y ago
      Github has a similar feature that’s free for public repositories IIRC.
    - Tiberium 2y ago
      https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog is a similar tool but checks for far more secrets, so I think it'd be a better choice.
- paxys 2y ago
  > If the incidents that made the rounds here in the last few months are any indication They really aren't. There's a huge world out there beyond the HN front page.
- AtlasBarfed 2y ago
  Another reason to be cloud agnostic. They are counting that switching cost will make you eat the bill. If you have to be cloud, do dev in one cloud and test/prod in another. I know, I know, easier said than done.
  - srockets 2y ago
    An argument that is often only made by folks who didn't ran anything at scale on a public cloud. If you've ever set in a room when a cloud deal was signed, you'd know it cable: you don't get to pay less by threatening to switch providers (I've seen people try to suggest that and get laughed out of the room), but by buying more from a single one. Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it. And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.
    - hinkley 2y ago
      In the IBM, HP days it wasn’t about paying less per se by threatening to switch providers, it was getting more concessions (of which money might be one). Some big companies had both systems so they could play them off of each other. Time to order more hardware, who will kiss our butts more? I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.
    - Aeolun 2y ago
      AWS is something like an insurance service. We all pay more so that they can forgive the people that mess up xD
      - srockets 2y ago
        That’s not how it works. We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances. As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.
- giraffe_lady 2y ago
  I have personal knowledge of two cases of this happening and both were immediately dropped. One was a high school student's personal project gone awry for like $3k worth the other was a startup where it was $120k or so. Two different providers. Neither had to particularly argue their case, just ask and wait a few days.
- jmathai 2y ago
  I was a PM in GCP and refund requests due to customer misconfigurations would make it to me for approval. Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary. Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.
  - rawgabbit 2y ago
    So no elastic anything. Just servers?
    - jmathai 2y ago
      If you have elasticity then make sure you understand the limits. Even Serverless has these knobs but for various reasons they default to high elasticity. But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.
- JimDabell 2y ago
  What are you referring to, exactly? That doesn’t sound like AWS at all. AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.
  - morningsam 2y ago
    You're right, I was getting two relatively recent posts mixed up: 1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1] 2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2] I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1). [1] https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776 [2] https://news.ycombinator.com/item?id=40203126 https://news.ycombinator.com/item?id=40203126
    - rmccue 2y ago
      For (2), AWS _also_ has now shipped a change that makes those requests non-billable permanently: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
- ascorbic 2y ago
  Well, the ones where they forgive it right away don't write blog posts about it, so you don't know about them. I guarantee that every case which makes the front page is either an example of a mistake made by an individual support engineer following the wrong script, or the person posting it is being dishonest about what actually happened (I recall a particularly suspicious recent one where they hid the name of their site). The cloud providers all have a policy of forgiving these sort of cases. The only speedbump is ascertaining if it was a genuine mistake.
- turok 2y ago
  We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.
  - mwarkentin 2y ago
    FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-recursive-loop-detection-apis/ https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...
    - turok 2y ago
      Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.
bdcravens 2y ago
To be clear, in the case of those abusing the policy: > We reserve the right to cut you off.
- Aeolun 2y ago
  While obvious, doesn’t this make it the same as before?
  - tptacek 2y ago
    The sentence he's referring to says that we might tell you after you've, uh, aggressively asked us to forgive a series of bills, that future accidents are no longer on the house. So no, it's not the same as before.
    - Aeolun 2y ago
      I say that because I’ve had extensive GPU bills credited or forgiven before. Presumably there was already a point where you were going to say “you keep messing up with the auto shut-off, at some point we’re going to stop refunding you” I’ll say it’s nice to have it explicit.
  - lo3p0ez 2y ago
    I trust them to handle even edgy cases more generous after this announcement. Just a gut feeling, as if they would risk losing significant credibility. I've written elsewhere that I'm not a big fan of those over-marketing posts, but they are small and more trustworthy than any of the large providers.
paxys 2y ago
It's unfortunate that the solution to cloud pricing complexity that all providers are adopting is – add even more complexity on top. The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be smoothened? None of this can be deterministically calculated from the pricing page. And there's no way for you to do these checks yourself before deployment because you have no idea what this logic even is. So you are entirely at the mercy of the provider for "forgiveness". Who wants to bet that some provider is going to launch "AI cloud billing" within the next year?
- bdcravens 2y ago
  It's far from perfect, but the large providers do give you tools for categorizing your charges (tagging, etc). There's some fuzziness, especially around data transfer, but for the most part developers can look at an application and know what the cost of each resource is. The biggest risk seems to be out-of-control auto scaling turned on without doing reasonable analysis first.
- candiddevmike 2y ago
  You'll pay exactly the amount they think you can afford to pay and not a penny less.
  - tptacek 2y ago
    That would be what you would pay in the absence of competition. This is an incredibly competitive space.
    - paxys 2y ago
      Competition only matters for new contracts. Once you have picked a provider and made a big enough infrastructure investment, there's no realistic path to switch to someone else.
tomjen3 2y ago
I really, really wish there was just a way to put a hard limit on how much a cloud could charge me. I am never going to want to spend 200k of my personal money on some project on a cloud. Never. I don't even want my ant-based basket viewing project simulator to cost me 1 thousand dollars because it went viral and all clouds overcharge for bandwith. Just let me put in a limit.
- cube2222 2y ago
  I think this sounds easy, but it’s actually quite nontrivial in practice. You need special handling of such a limit for each kind of resource. E.g. when my limit is reached to they remove the database, along with all backups, and all objects on S3? Since storage is billed, it should also be stopped when the limit is reached, right? I think in practice among companies paying most of their revenue, there’d be zero interest in this, while it would be a lot of effort to implement. This is really something specific to hobby projects, which just shouldn’t be using those “unlimited potential cost” services.
  - candiddevmike 2y ago
    I don't think any of these requirements are necessary. A basic "if I reach my bill limit, turn off things that are billing" toggle would suffice for 80%+ of users, especially with better billing controls/per-team billing accounts. I think you run into more problems/user frustrations with a tenuous conditional-shut-off approach. My tinfoil hat is that a lot of cloud billing is accidental, probably from "lab environments", and they don't want to provide a way to budget/limit these.
    - tptacek 2y ago
      The whole post is about this accidental billing situation.
    - cube2222 2y ago
      I mean, those S3 objects, those backups, those databases, are billing. “Turning them off” means deleting them.
scop 2y ago
One of the things I really admire about Fly is "how human they come across". Fly is a collection of people providing a computing service to other people. As a fellow human, there is a very primitive part of my brain that will always prefer "tech from humans" over "tech". Given that the very teleology of cloud computing is digital & ephemeral resources, having an actual human face associated with it makes it tangible in a way that is hard to place. My little brain can briefly understand complex computer systems on a need-to-know basis, with knowledge constantly coming and going based on the demands of the day. I can never fully "understand" my cloud infrastructure. But hey, Kurt's standing there. I like Kurt. I can understand Kurt as he's a human and so am I and he's in the same exact place I am. Let's work hard and make cool stuff, what do ya say Kurt?
- scop 2y ago
  I should add that another way Fly distinguishes itself as a "tangible" cloud company is that their blog art is all based on physical mediums. It feels very familiar and quantifiable to my brain.
  - tptacek 2y ago
    That's Annie Ruygt, our in-house illustrator. Check out her work; she's amazing. https://annieruygtillustration.com/ https://annieruygtillustration.com/
gwbas1c 2y ago
> and may only come within tens of dollars of accurately predicting your water bill. Whenever I've had to water new grass seed I've always been surprised by my water bill.
- hinkley 2y ago
  There’s a guy on Reddit a couple days ago showing off his softball sized watermelon he grew with a $100 watering bill. The struggle is real.
  - aidenn0 2y ago
    $100 is like 14HCF where I live (though it includes sweage). [edit] 14, not 12HCF.
    - hinkley 2y ago
      That’s the problem with irrigation. You get charged for the sewage cost of that much water even though it doesn’t end up there. Unless you’re a farmer.
- tptacek 2y ago
  There's a note on the draft of this post with like 6 comments from people on our team about how the water bill is a bad example. I stand by what I wrote!
- genewitch 2y ago
  my well only costs me electricity and has no meter - well, i have a meter somewhere, i just never installed it. I know for a fact it uses 9A to run, so it would cost me 30¢ or so, all-in, to run it for an hour. it's somewhere around 10-12GPM, so 600-700 gallons for 30c I've wanted to put it on solar the entire time i've had it but the start current is 18A and an inverter that can handle that is (or maybe was, idk) real expensive, considering. anyhow if you saw the area around my house, in the dog days of summer, right now, you'd think it had been raining non-stop all summer - in fact, it hasn't really rained at all since the beginning of june when we had 12" - i've personally watered 6+" over about 1/3rd of an acre, so like 50,000 gallons since then. edit: oh crap i forgot the reason i wanted to comment on this at all! My neighbor recently had some issues with a pipe running along the bottom of his house that sprung a leak and they got a surprise $600 water bill - no forgiveness. the water company did, in fact, have the audacity to tell my neighbor "there's a light on the meter if you have a leak." His meter is like a quarter mile from his house, first of all. and second of all, when they read the meter and saw the light, why didn't they go across the street and knock on his door, or make a note to call/mail? I'll tell you why. $600
aeternum 2y ago
New startup idea: Pool unused "accident" credits, automate the forgiveness request, and sell it to the highest bidder.
- datadrivenangel 2y ago
  I believe that's called a reserved instance.
joshfraser 2y ago
In other words, they are socializing the costs. Servers and electricity aren't free. Wouldn't it be better for the customer if they had no accident forgiveness and passed those cost savings along? Instead, you are paying for other peoples mistakes plus all the extra overhead caused from fraud that they incentivized.
- tptacek 2y ago
  No, it would be worse if we "passed these cost savings along".
- trjordan 2y ago
  The servers are already bought; the electricity cost is negligible for actual accidents. As mentioned in the post: the hosting providers don't actually pay marginal costs for transient mistakes, so neither should honest customers.
  - joshfraser 2y ago
    And what about the extra costs fighting fraud now that they've advertised this policy?
    - DAlperin 2y ago
      We have to invest in fighting fraud and abuse anyway, such is the public cloud business. We don't intend to diminish user experience in service of fighting it.
osbulbul 2y ago
the biggest reason of I am not using aws, google cloud, vercel etc... for my personal projects is surprise bills. I am not earning anything from them so I can only put $50 but can't afford $500 or $5000. so still i will feel much more secure if I can put hard limit and absolutely sure my bill will never go above $50. (or cloud billing insurance :)) but you got my attention, i can try fly.io
- tptacek 2y ago
  Except for things like student accounts, which we're working on, I don't think hard limits are coming any time soon. Our expectation is that the enforcement of hard billing limits would mostly make customers furious with us. If you read to the end of the post, the direction we're going is preemptive detection of weird billing spikes, so you don't even have to notice and ask us. If you're really only looking to spend $50, we should put our cards on table and say that we're generally not making product pricing decisions with you in mind. If your needs are pretty straightforward, there are hosting providers that will do a better job of serving that business than we will.
  - osbulbul 2y ago
    yeah I am pretty sure no cloud provider thinks people who can willing to spend small money. but guess what? if one of my projects start earning money, then I will continue to what I know best :) I think that's why every cloud provider still has configs for couple of $
    - tptacek 2y ago
      We think all the time about people starting small projects here. I think a really good line to draw, if you want to understand how we think about this stuff, is between people who would be OK with us terminating their service because they mispredicted what their cap should be, and people who need their stuff to stay up and running and will just talk to support if they're concerned about billing. A shorter way to say this is that we making product pricing decisions for people who are doing this work professionally.
Poiesis 2y ago
Thomas, every time I read something you write it's a delight. I love your writing style, and it reads to me like you're always putting effort into making it succinct yet unambiguous, without unnecessary embellishment.
- tptacek 2y ago
  The trick is just that I write specifically for this place.
bitbasher 2y ago
There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.
- klabb3 2y ago
  Yes, but I imagined that you’d have a ceiling for rolling average / token bucket over a sensible time window, with two limits: - a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out” - a hard limit which fails until more tokens trickle back in, without shutting down service I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing. Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought
- tptacek 2y ago
  I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.
  - sigseg1v 2y ago
    For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than 10% of my monthly spend if left running for 24h." This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.
deleted 2y ago
[deleted]
lucifargundam 2y ago
I just wanted to say I first heard of Fly.io from "The Changelog" podcast
deleted 2y ago
[deleted]
brundolf 2y ago
I've never understood why providers don't just give you the option of setting a cost cap: "If this piece of infra exceeds [5x expected usage cost], shut it down and send me an email"
- lo3p0ez 2y ago
  Deno recently did: https://deno.com/blog/deploy-spend-limits https://deno.com/blog/deploy-spend-limits
veggieroll 2y ago
This thread is full of people complaining that they just want the ability to set a hard billing cap. And yet, providers continue to insist that no serious customer wants this and/or it's impossible to implement. What would it take for providers to listen to real customers here? I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If you want my $25k, I need to ensure that I don't spend more than this amount. As is, my solution is to use old-school pre-rented, long-term contracted commitment VM hosting providers. This is really the only way to guarantee that you are paying an exact amount and no more. But, I really would like to use a more scalable system that didn't require pre-provisioning. And, I wish people would believe the customer when they say something and not continue to gaslight us. Providers say it's impossible, but I don't see how it would be so hard. Here's my sketch of how it could work: The main component is a system that monitors billing events and watches for the slope of the bill to ensure that there is enough runway to stay under the cap. Optionally, they could implement rate limits on resource creation to ensure that a sudden surge doesn't outrun the monitoring. You also need notifications for when the projected spend exceeds the cap. Optionally, you could implement a soft cap, where no new resources can be created. And finally, you need the hard cap where things start to get deleted. If they're feeling generous, the provider could implement a period where VM's/lambda's/etc are shut off, blob storage is not accessible, and so on so that the account holder has some time to fund the account and/or fix whatever is causing the overage. That set of features are all totally within the competency of a cloud provider. Knowing how much things cost, billing for them, and turning them on and off is their main business. And I can't believe that they expect us to believe that it's impossible to do that tracking. This is how I do system migrations. There are escalating warnings until one day the service is shutoff for an hour and turned back on. That wakes up any laggards that missed the dozen or so communications over a three-to-six month period. Finally, after a few more days, the service is shutdown for good and then data is deleted a week after that. Though I almost always keep a copy in cold storage. But that isn't necessary as a provider with a limited relationship to the client.
- tptacek 2y ago
  Nobody's gaslighting you. It's not impossible to build this, though it is much more difficult than it seems (cloud billing is a large-scale eventually-consistent distributed system, and if you've done any distributed systems work the issue with plugging a system like that directly into a control loop should be obvious). It's just expensive to build, and disproportionately serves the interests of customers who aren't running real apps. At $25k/mo spend, you can talk to many cloud providers (certainly including us) to work out an "I can't get invoiced for more than $25k" solution which will not involve having your app turned off abruptly when the 2,500,001st cent get spent in October. What you'll notice in this thread is that people generally want billing caps for accounts they plan to spend, like, $10/mo on. And we can build cap systems for those people --- but they'll involve turning parts of the platform off for them. I'm really pleading with people with strong opinions about hard caps to do the exercise of working out how these billing systems work. There are a huge number of apps running here, across a huge fleet of physicals, running in almost 40 regions around the world. Each of those apps has several different kinds of resources that meter at different granularities and incur different costs. Speaking as a witness to the creation of a new billing system just a month or two ago: it is kind of a miracle that these things work at all. Do the thought experiment, read some Call Me Maybe posts, and then tell me it's obvious that this feature should be straightforward to build.
theusus 2y ago
Their docs are all over the place. I don't understand what their default offerings are. And that makes it hard for me to chose them.
- tptacek 2y ago
  What do you mean by "default offerings"? Thanks!
  - theusus 2y ago
    https://www.heroku.com/ https://www.heroku.com/ Checkout their products
    - taormina 2y ago
      Heroku is not Fly.io. Heroku was bought by the MBAs a while back, and they do something different than generic cloud hosting (they are closer to a hosted EBS). If you bother scrolling at all, you'll notice immediately below the fold that the pricing is not remotely as clear as you're pretending it is.
tptacek 2y ago
Y'all really, really want caps. We think that's a crazy feature. We don't want to do it. But I just had this conversation on a Slack with a bunch of friends at other companies and they were just as yell-y about this. There's a threshold of feedback at which I think you could get us to do some kind of cap thingo. I'm just saying. Yeesh.
- ustad 2y ago
  In your article or in the posts here you have not said why you think its a crazy idea.
  - tptacek 2y ago
    I've said it over and over again: if you have a serious app, and someone somehow steals a credential from you and uses it to light up a bunch of crypto miners, you don't want us shutting down your main app in response. We perceive it mostly as a feature that will blow people up. We agree about the underlying problem! You don't want to spend $5000 in a month for services you never wanted. We don't want you spending that either. We'd rather just improve our billing so that you can fix this after the fact without trading off availability. But I have just planted a flag: we are prepared to be wrong about this. I don't think we are, but like, I'm the only one. :)
    - ustad 2y ago
      Crypto miners is the reason for not having a “crazy” option for “cap on/off” at sign up? You know what you can’t fix after the fact of getting a huge bill? a heart attack.
      - tptacek 2y ago
        This post is about us not giving you that heart attack. I'm pushing back, and I'm going to keep pushing back; if we do this feature, it is going to be kicking and screaming. :)
    - bee_rider 2y ago
knowitnone 2y ago
no matter what, you'll have some fraud. Perhaps a better solution is the customer has a know they can turn that sets max limits and some alerts on when usage is abnormal.
csours 2y ago
> " You probably can’t tell me how much electricity your home is using right now, and may only come within tens of dollars of accurately predicting your water bill. But neither of those bills are all that scary, because you assume there’s a limit to how much you could run them up in a single billing interval. " I had a $600 surprise water bill. It was (partially) forgiven because the water department could drive to my house and see evidence of the leak next to my water meter. It did turn out to be on my side of the meter, so it is my responsibility. If the water department had driven to my house and seen evidence of commercial agriculture (so to speak), then it would not have been forgiven. --- The parallel here is that the water department can't come into my house uninvited - the cloud provider SHOULD NOT have intimate access to the running code, but they are able to observe some patterns without 'breaking in'. --- Side note: the size of the bill and the amount of forgiveness was largely driven by waiving an 'excess usage' surcharge - similar to how you can get a discount for cloud service reservations.
- wrs 2y ago
  Water billing is indeed nonlinear with no cap. I had a surprise $3000 excess water bill due to a broken connection at the meter. Followed by a $3000 excess sewer bill (!) because the assumption is if I used $3000 of water, it must have gone down the drain. However, if you demonstrate to an inspector that the water must have leaked below ground (as in this case) there’s a process for getting the second charge waived. Unfortunately the leak was on our side of the meter, so the first charge was correct, though we got partial forgiveness.
  - genewitch 2y ago
    So, when they read the meter, and saw that it was flooded at the meter or nearby, they didn't think to mention that to you? I ask because you said "surprise bill" - another comment i made explains my neighbor's water meters have a light on them that lights up if there's a leak, which the meter reader can see and the homeowner usually cannot - the only reason to not notify the homeowner of an issue is because of $.
    - 2y ago
ctennis1 2y ago
Google Cloud will not cut you a break unless you have an incredible about of pull. Even then, you won't get much of one.
- smithcoin 2y ago
  I'm suffering for this now. They just erroneously charged me almost $4,000 out of nowhere because they decided to scan all of old container images for no reason and without notification. I have a support partner and everybody is just passing the buck around, it's super frustrating.
- aidenn0 2y ago
  Other comments here (e.g. https://news.ycombinator.com/item?id=41333600 https://news.ycombinator.com/item?id=41333600) suggest that this is not universally true.
- campers 2y ago
  Being at the level where you have an account manager should help. Our technical account manager always said if we ever accidentally rack up a bunch of cloud costs we should at always ask if we can get a refund, and as long as its infrequent there's a good chance to get some credits.
spencerchubb 2y ago
When I was in highschool, my friends and I were doing a small project. My friend accidentally ran Google's image recognition service in a while loop for a whole minute, and racked up a $300 bill. Thankfully GCP waived it for us.
- reaperman 2y ago
  I ran up a $4,000 bill with like a week of TPU time bc I didn’t realize that they kept running after the attached CPU turned off (unlike GPU offerings). GCP also waived the whole thing - even refunding the $4k they had already pulled from my payment method.
  - spencerchubb 2y ago
    That's brutal, and if you're anything like me, probably learned a lesson very thoroughly for the future!
ultra_nick 2y ago
Build everything on Docker, Kubernetes, and VPS. Run on the cheapest VPS provider with human support. Make a quick and easy switch if they get too shifty.
andrewstuart 2y ago
Just run Linux computers on ionos with unlimited traffic and get on with the job instead of worrying about silly things like how much traffic passed over some internal software system and will it bankrupt the company. https://www.ionos.com/servers/vps https://www.ionos.com/servers/vps
- ilrwbwrkhv 2y ago
  Same but on Hetzner. Incredibly beefy machines. Unlimited traffic. No fear of any sudden changes that I explicitly didn't act upon. I do not know why people fall for the shiny marketing so easily, that too hackers? Is it all the Javascript folks? Vercel I know is propped up by them for sure.
  - tptacek 2y ago
    I am never going to get tired of telling people that if they can solve their problems running on Hetzner, they should do that. It's not like a dirty secret of our industry. By all means, deploy with VPSs! Or on bare metal servers!
    - ilrwbwrkhv 2y ago
      Do you have a blog post on fly's blog where you tell people that?
      - tptacek 2y ago
        We are incredibly backlogged on the blog. I've got 3000 words on Tigris rolling out next week, we've got the results of a $300K 3rd party GPU audit to write up, we wrote a replacement to Vault called Pet Semetary, we need to write up the billing system because, as you can see from this thread, people appear to think that's not a hard technical problem. I can imagine getting around to writing a post about when a VPS is a good alternative to launching a container on Fly.io, but it'll be aspirational; I doubt I'll ever get around to it. But I'm happy to remind people here of it!
      - 2y ago
aidenn0 2y ago
IANAF, but Lower Montana seems like a terrible place to farm coffee beans?
- maverwa 2y ago
  So Kurt is going to hate that job, too?
- tptacek 2y ago
  I asked him about this and he didn't explain, but did demand that I keep it. He has a weird thing about Montana.
Aeolun 2y ago
> may only come within tens of dollars of accurately predicting your water bill Uh, what is the magnitude of these water bills? I’m fairly certain I can predict it to within a few dollars because it’s only $35 in the first place.
- lupire 2y ago
  My water bill is $200/quarter.
- School-Cotton 2y ago
  I have absolutely no idea how much I pay for water, but vaguely think it’s probably between 0 and 100. Why does knowing it’s small mean you can predict it precisely, in a world where auto-payment exists so you never have to actually look at your bills?
  - Aeolun 2y ago
    I mean that it’d be extremely unlikely for me to not be able to predict my water bill to within ‘tens of dollars’, because that is the entire amount. Also, I get receipts by postal mail even if I auto pay them, so I guess that makes it easier.
srazzaque 2y ago
Whilst it doesn't have the mindshare or some features which are tablestakes for enterprise customers, I've found Linode's pricing to be extremely predictable for small projects. Even post Akamai acquisition. I'm sure other smaller players are also just fine. Couple that with the fact you can achieve quite a bit on simple set-ups that are adequately sized to begin with, you can save quite a bit. Not all of us need elasticity, or environments being spun up/down on commit.
tonymet 2y ago
It's important that all utility metering and pricing is effectively made up to balance costs across customers. That's why residents in california probably pay 100000x / unit of water than agriculture customers do . The same works for the cloud. An operation that costs $100000 does not cost the operator $85000 in operations on the margins. Yes the metering and pricing schedules are necessary, but punitive pricing for accidents really is just an artifact of the system. I'm guessing the biggest reason AWS provides forgiveness is that if anyone took them to court over a bill, a court would throw out the charge once the wholesale cost was revealed.
kasey_junk 2y ago
If I were fly I’d implement spending caps just to filter out unserious operators. My implementation of that feature would be when you turned it on fly would just kick you off. No serious operator wants a provider to turn their system off. So if you want that it’s pretty clear you are hosting a silly system. Which likely costs more to support and drives less margin.
- _jab 2y ago
  No serious operator would choose a provider with your implementation of that feature.
  - tptacek 2y ago
    I agree that no serious operator would choose a provider whose spending cap feature was "enable this feature and we kick you off the platform".
    - psd1 2y ago
      People will pay to be abused: c.f. Oracle
- lo3p0ez 2y ago
  Hu? For any business having a due to a cap throttled service is better than losing the money straight away due to an accidental cloud bill explosion.
- icedchai 2y ago
  The real solution is to make it simpler to set up billing alerts. If my hobby project is ever projected to cost more than $0.50/hr, let me know. Billing alerts should be part of onboarding for all cloud providers, especially for individuals that don’t want surprises.
- langcss 2y ago
  And yet every single cloud let's you scale from zero and play and experiment. They know today's tinkerers are tomorrow's CTO decision makers.
lo3p0ez 2y ago
[flagged]
jowdones 2y ago
That's why my phone SIM is still not a subscription but rather I manually buy credit each month. The price is even lower than subscription and benefits exceed it, shitloads of Gb and minutes. If I'll use roaming I'll pay for a more expensive credit that month but I have piece of mind I won't be charged thousands of euro at the end of the month. Like telecom providers of course cloud providers could have metered service billed under "get only as much as you paid" policy but it's obviously they totally and in bad faith don't want that.
figassis 2y ago
This is why I love Hetzner. It has billing alerts. What was so difficult about it that fly had to rebuild their system? Also, do everything except databases inside kubernetes. Deploy kubernetes across multiple clouds via wireguard. Label your instances properly on each provider. Prefer bare metal instances where available. Migrate your workloads accordingly. Force cloud providers to earn your money. Don’t however have both workloads running at the same time in multiple providers as you will eat insane data costs.
- psd1 2y ago
  I can't read the tone, but perhaps you haven't seen much technical debt. It's possible to have a fairly good codebase, full of good decisions, yet you still can't "get there from here". Second paragraph was useful, thanks!
campers 2y ago
> Meanwhile: like every public cloud, we provision our own hardware, and we have excess capacity. Your messed-up CI/CD jobs didn’t really cost us anything This^ Don't feel bad about asking for credits when you accidentally make a costly mistake.
nottorp 2y ago
Isn't it easier to set hard limits on the account and notify the client when they're close to them so they can decide to increase them on their own if needed? But then you run into the risk that the client will never increase those hard limits and ... pay less than they could. Not good for business.
bruce343434 2y ago
So AI is now being used on the Cloud to fight the Blockchain?
deleted 2y ago
[deleted]
lolinder 2y ago
> The Fully Automated Accident-Forgiving Billing System of the Future (which we are in fact building and may even one day ship) will give you a line-item veto on your invoice. For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over budget than ever see a bill that's higher than our net worth. A line item veto somewhat alleviates that concern but not fully. They indicate at the end that they're going to do something along these lines, but what they're describing there also seems over-engineered compared to a simple circuit breaker that kills the account or some subset of it. Is there a good reason for these providers to avoid implementing that feature, which on a naive look seems far simpler than what they've actually proposed?
- tptacek 2y ago
  Something like half the comments on this story are a discussion of why or why not cloud providers do or don't provide this simple circuit breaker feature.
  - lolinder 2y ago
    I read yours after posting this and it provides exactly zero information: > I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. "Terminating service is a big deal" how? I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend? No one is asking for automatic circuit breakers applied to all customers indiscriminately, but I'm not seeing any justification in here for why an opt-in circuit breaker is technically or legally challenging to implement.
    - kikoreis 2y ago
      > I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend? As someone also involved in billing systems for public clouds: in theory there's no difference, but in practice there is a world of difference. This is the sort of situation where the end user is commonly surprised with the consequences of their own decisions. At MGC we have some "soft shut down" processes, and we constantly hear stuff like "I know I said shut down, but this is the one situation where that really didn't make sense"; where examples are "storage which keeps backups became unavailable", "a very simple but critical user auth system disappeared", "I had no idea this was still running on my account", or "OMG not in the middle of the weekend", etc. You can build heuristics and tracking into the system to minimize these situations, but that's a lot of work. So yeah, it is a valid use case and something many CSPs would like to provide, but implementing something that is actually better than nothing is non-trivial.
timoxley 2y ago
I feel like this indicates there's a market for traffic-spike insurance

310 points by piperswe 2y ago

lagniappe 2y ago

This shouldn't be a selling point, it should just be table stakes.

sealeck 2y ago
You can say this for a whole bunch of new features people role out but this doesn't erase the fact that they are very much not the stakes and introducing them is great because it raises them!
tptacek 2y ago
We agree.

morningsam 2y ago

>If you do something luridly stupid and rack up costs, AWS and GCP will probably cut you a break. [...] Everyone does. If the incidents that made the rounds here in the last few months are any indication, they'll start out insisting you pay no matter what. You'll then have to write a blog post about it, post it to Twitter, HN, and Reddit, get a couple hundred comments expressing anger at the provider, and wait for someone from their PR department to see it. Only then will they finally waive the costs. Good on Fly.io for trying to handle such situations more sensibly.

bdcravens 2y ago
There's plenty of situations you don't hear about. A few years ago I f'ed up and accidentally pushed keys to a public repo, and by the next morning, we racked up $50k in AWS charges from crypto miners. We reached out, they gave us a security checklist that if we followed, they'd take off the charges. We did, and by Monday (my code push was Friday evening) the charges were taken off. No public shaming required.
- paulgb 2y ago
  I’m curious what sorts of things were on the checklist. I wonder if it’s something they will share proactively?
  - bdcravens 2y ago
    It's been a few years, so I'm going off of memory, but it was mostly best practices stuff (enabling Cloudtrail, rotating older keys, etc). Anything to ensure that once the attackers no longer had access, removing/monitoring anything that would have longer term implications.
    - skeeter2020 2y ago
      this makes sense; plug the hole and stop taking on water before you ask for a free cleanup.
- perchlorate 2y ago
  You likely already know that, but to anyone else interested: a good way to prevent these kinds of situations is to run 'nosey parker' on your git repo before pushing it to a remote. It will dig through your code and configs, looking at files and through all the git history, and highlight anything that looks like tokens, passwords, keys, etc. You can set it as a pre-commit hook to block the offending code from even being committed. https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
  - Aeolun 2y ago
    Github has a similar feature that’s free for public repositories IIRC.
  - Tiberium 2y ago
    https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog is a similar tool but checks for far more secrets, so I think it'd be a better choice.
paxys 2y ago
> If the incidents that made the rounds here in the last few months are any indication They really aren't. There's a huge world out there beyond the HN front page.
AtlasBarfed 2y ago
Another reason to be cloud agnostic. They are counting that switching cost will make you eat the bill. If you have to be cloud, do dev in one cloud and test/prod in another. I know, I know, easier said than done.
- srockets 2y ago
  An argument that is often only made by folks who didn't ran anything at scale on a public cloud. If you've ever set in a room when a cloud deal was signed, you'd know it cable: you don't get to pay less by threatening to switch providers (I've seen people try to suggest that and get laughed out of the room), but by buying more from a single one. Hence accident forgiveness is in the same marketing bucket as free credits for new customers: reducing the aversion to trying the cloud and putting more work on it. And the switching costs, the biggest line item isn't building for multiple clouds (you can't be cloud agnostic: abstractions always leak somewhere, so you need to select a set of specific clouds to build for), but the cost of moving data between clouds. That's the real lock-in.
  - hinkley 2y ago
    In the IBM, HP days it wasn’t about paying less per se by threatening to switch providers, it was getting more concessions (of which money might be one). Some big companies had both systems so they could play them off of each other. Time to order more hardware, who will kiss our butts more? I’d be very surprised if that doesn’t still exist for the big boys. Though most of us are not big boys, and half of the biggest boys are cloud providers themselves.
  - Aeolun 2y ago
    AWS is something like an insurance service. We all pay more so that they can forgive the people that mess up xD
    - srockets 2y ago
      That’s not how it works. We all pay more [sic] so data centers would be over provisioned, allowing quickly expansion when we need it. I’ve been in a few incident that were root caused to a cloud provider lacking capacity to provision more instances. As that over capacity would’ve been bought and installed in the cloud provider’s data centers regardless of the errors that are being forgiven, not billing for those errors is a net gain to the provider, at minuscule cost, if at all, to other users.
giraffe_lady 2y ago
I have personal knowledge of two cases of this happening and both were immediately dropped. One was a high school student's personal project gone awry for like $3k worth the other was a startup where it was $120k or so. Two different providers. Neither had to particularly argue their case, just ask and wait a few days.
jmathai 2y ago
I was a PM in GCP and refund requests due to customer misconfigurations would make it to me for approval. Generally, I tried to make exceptions to grant the refund. It sucks to get these bills and they can be quite scary. Unless you have a real chance of bursty traffic I suggest going with compute that has more predictable costs.
- rawgabbit 2y ago
  So no elastic anything. Just servers?
  - jmathai 2y ago
    If you have elasticity then make sure you understand the limits. Even Serverless has these knobs but for various reasons they default to high elasticity. But many folks don’t need the elasticity at all. So you should factor that into your architectural decisions.
JimDabell 2y ago
What are you referring to, exactly? That doesn’t sound like AWS at all. AWS are very well-known for bill forgiveness. It’s not something set in stone, but if your bill explodes accidentally, even due to a mistake you made, they will normally forgive it if you ask them. You don’t need to go running to social media at all.
- morningsam 2y ago
  You're right, I was getting two relatively recent posts mixed up: 1. After a DDoS attack, someone got a $100k bill from Netlify for his static site and after he asked to have it waived, they generously reduced it to $5k. Only after his posts about it blew up did Netlify waive it completely. [1] 2. Someone got a $1k bill from AWS because lots of people made _unauthorized_ requests against his empty S3 bucket. AWS did agree to waive it immediately, prior to any social media posts. [2] I probably just remembered (2) as "that ridiculous billing situation involving AWS" but got the details of what exactly happened mixed up with (1). [1] https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776 [2] https://news.ycombinator.com/item?id=40203126 https://news.ycombinator.com/item?id=40203126
  - rmccue 2y ago
    For (2), AWS _also_ has now shipped a change that makes those requests non-billable permanently: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
ascorbic 2y ago
Well, the ones where they forgive it right away don't write blog posts about it, so you don't know about them. I guarantee that every case which makes the front page is either an example of a mistake made by an individual support engineer following the wrong script, or the person posting it is being dishonest about what actually happened (I recall a particularly suspicious recent one where they hid the name of their site). The cloud providers all have a policy of forgiving these sort of cases. The only speedbump is ascertaining if it was a genuine mistake.
turok 2y ago
We incurred a 6 figure bill when the API an authentication token handling lambda was updating from was taken down. The lambda went into a crazy loop self invoking, as it had a retry mechanism and a CRON schedule, which piled invocations on top of retries. (over worked team, poor design, etc.) So far we have gotten no concessions from AWS, and we have annual bills in the millions, just not for this application whose budget now has an awkward and obvious spike.
- mwarkentin 2y ago
  FWIW there are protections for this now: https://aws.amazon.com/blogs/compute/aws-lambda-introduces-recursive-loop-detection-apis/ https://aws.amazon.com/blogs/compute/aws-lambda-introduces-r...
  - turok 2y ago
    Thanks, we are trying to compose an argument with this service in mind, we didn't have any recursive invocation protections for this lambda, and the AWS services it hammered during its invocations contributed to the massive cost.

bdcravens 2y ago

To be clear, in the case of those abusing the policy: > We reserve the right to cut you off.

Aeolun 2y ago
While obvious, doesn’t this make it the same as before?
- tptacek 2y ago
  The sentence he's referring to says that we might tell you after you've, uh, aggressively asked us to forgive a series of bills, that future accidents are no longer on the house. So no, it's not the same as before.
  - Aeolun 2y ago
    I say that because I’ve had extensive GPU bills credited or forgiven before. Presumably there was already a point where you were going to say “you keep messing up with the auto shut-off, at some point we’re going to stop refunding you” I’ll say it’s nice to have it explicit.
- lo3p0ez 2y ago
  I trust them to handle even edgy cases more generous after this announcement. Just a gut feeling, as if they would risk losing significant credibility. I've written elsewhere that I'm not a big fan of those over-marketing posts, but they are small and more trustworthy than any of the large providers.

paxys 2y ago

It's unfortunate that the solution to cloud pricing complexity that all providers are adopting is – add even more complexity on top. The number you see on your bill is increasingly calculated by running some black box algorithm on top of the billing events your resources generate. Was it accidental or not? What is a "weird" deployment vs a normal deployment? By what factor should the spikes on your billing graph be smoothened? None of this can be deterministically calculated from the pricing page. And there's no way for you to do these checks yourself before deployment because you have no idea what this logic even is. So you are entirely at the mercy of the provider for "forgiveness". Who wants to bet that some provider is going to launch "AI cloud billing" within the next year?

bdcravens 2y ago
It's far from perfect, but the large providers do give you tools for categorizing your charges (tagging, etc). There's some fuzziness, especially around data transfer, but for the most part developers can look at an application and know what the cost of each resource is. The biggest risk seems to be out-of-control auto scaling turned on without doing reasonable analysis first.
candiddevmike 2y ago
You'll pay exactly the amount they think you can afford to pay and not a penny less.
- tptacek 2y ago
  That would be what you would pay in the absence of competition. This is an incredibly competitive space.
  - paxys 2y ago
    Competition only matters for new contracts. Once you have picked a provider and made a big enough infrastructure investment, there's no realistic path to switch to someone else.

tomjen3 2y ago

I really, really wish there was just a way to put a hard limit on how much a cloud could charge me. I am never going to want to spend 200k of my personal money on some project on a cloud. Never. I don't even want my ant-based basket viewing project simulator to cost me 1 thousand dollars because it went viral and all clouds overcharge for bandwith. Just let me put in a limit.

cube2222 2y ago
I think this sounds easy, but it’s actually quite nontrivial in practice. You need special handling of such a limit for each kind of resource. E.g. when my limit is reached to they remove the database, along with all backups, and all objects on S3? Since storage is billed, it should also be stopped when the limit is reached, right? I think in practice among companies paying most of their revenue, there’d be zero interest in this, while it would be a lot of effort to implement. This is really something specific to hobby projects, which just shouldn’t be using those “unlimited potential cost” services.
- candiddevmike 2y ago
  I don't think any of these requirements are necessary. A basic "if I reach my bill limit, turn off things that are billing" toggle would suffice for 80%+ of users, especially with better billing controls/per-team billing accounts. I think you run into more problems/user frustrations with a tenuous conditional-shut-off approach. My tinfoil hat is that a lot of cloud billing is accidental, probably from "lab environments", and they don't want to provide a way to budget/limit these.
  - tptacek 2y ago
    The whole post is about this accidental billing situation.
  - cube2222 2y ago
    I mean, those S3 objects, those backups, those databases, are billing. “Turning them off” means deleting them.

scop 2y ago

One of the things I really admire about Fly is "how human they come across". Fly is a collection of people providing a computing service to other people. As a fellow human, there is a very primitive part of my brain that will always prefer "tech from humans" over "tech". Given that the very teleology of cloud computing is digital & ephemeral resources, having an actual human face associated with it makes it tangible in a way that is hard to place. My little brain can briefly understand complex computer systems on a need-to-know basis, with knowledge constantly coming and going based on the demands of the day. I can never fully "understand" my cloud infrastructure. But hey, Kurt's standing there. I like Kurt. I can understand Kurt as he's a human and so am I and he's in the same exact place I am. Let's work hard and make cool stuff, what do ya say Kurt?

scop 2y ago
I should add that another way Fly distinguishes itself as a "tangible" cloud company is that their blog art is all based on physical mediums. It feels very familiar and quantifiable to my brain.
- tptacek 2y ago
  That's Annie Ruygt, our in-house illustrator. Check out her work; she's amazing. https://annieruygtillustration.com/ https://annieruygtillustration.com/

gwbas1c 2y ago

> and may only come within tens of dollars of accurately predicting your water bill. Whenever I've had to water new grass seed I've always been surprised by my water bill.

hinkley 2y ago
There’s a guy on Reddit a couple days ago showing off his softball sized watermelon he grew with a $100 watering bill. The struggle is real.
- aidenn0 2y ago
  $100 is like 14HCF where I live (though it includes sweage). [edit] 14, not 12HCF.
  - hinkley 2y ago
    That’s the problem with irrigation. You get charged for the sewage cost of that much water even though it doesn’t end up there. Unless you’re a farmer.
tptacek 2y ago
There's a note on the draft of this post with like 6 comments from people on our team about how the water bill is a bad example. I stand by what I wrote!
genewitch 2y ago
my well only costs me electricity and has no meter - well, i have a meter somewhere, i just never installed it. I know for a fact it uses 9A to run, so it would cost me 30¢ or so, all-in, to run it for an hour. it's somewhere around 10-12GPM, so 600-700 gallons for 30c I've wanted to put it on solar the entire time i've had it but the start current is 18A and an inverter that can handle that is (or maybe was, idk) real expensive, considering. anyhow if you saw the area around my house, in the dog days of summer, right now, you'd think it had been raining non-stop all summer - in fact, it hasn't really rained at all since the beginning of june when we had 12" - i've personally watered 6+" over about 1/3rd of an acre, so like 50,000 gallons since then. edit: oh crap i forgot the reason i wanted to comment on this at all! My neighbor recently had some issues with a pipe running along the bottom of his house that sprung a leak and they got a surprise $600 water bill - no forgiveness. the water company did, in fact, have the audacity to tell my neighbor "there's a light on the meter if you have a leak." His meter is like a quarter mile from his house, first of all. and second of all, when they read the meter and saw the light, why didn't they go across the street and knock on his door, or make a note to call/mail? I'll tell you why. $600

aeternum 2y ago

New startup idea: Pool unused "accident" credits, automate the forgiveness request, and sell it to the highest bidder.

datadrivenangel 2y ago
I believe that's called a reserved instance.

joshfraser 2y ago

In other words, they are socializing the costs. Servers and electricity aren't free. Wouldn't it be better for the customer if they had no accident forgiveness and passed those cost savings along? Instead, you are paying for other peoples mistakes plus all the extra overhead caused from fraud that they incentivized.

tptacek 2y ago
No, it would be worse if we "passed these cost savings along".
trjordan 2y ago
The servers are already bought; the electricity cost is negligible for actual accidents. As mentioned in the post: the hosting providers don't actually pay marginal costs for transient mistakes, so neither should honest customers.
- joshfraser 2y ago
  And what about the extra costs fighting fraud now that they've advertised this policy?
  - DAlperin 2y ago
    We have to invest in fighting fraud and abuse anyway, such is the public cloud business. We don't intend to diminish user experience in service of fighting it.

osbulbul 2y ago

the biggest reason of I am not using aws, google cloud, vercel etc... for my personal projects is surprise bills. I am not earning anything from them so I can only put $50 but can't afford $500 or $5000. so still i will feel much more secure if I can put hard limit and absolutely sure my bill will never go above $50. (or cloud billing insurance :)) but you got my attention, i can try fly.io

tptacek 2y ago
Except for things like student accounts, which we're working on, I don't think hard limits are coming any time soon. Our expectation is that the enforcement of hard billing limits would mostly make customers furious with us. If you read to the end of the post, the direction we're going is preemptive detection of weird billing spikes, so you don't even have to notice and ask us. If you're really only looking to spend $50, we should put our cards on table and say that we're generally not making product pricing decisions with you in mind. If your needs are pretty straightforward, there are hosting providers that will do a better job of serving that business than we will.
- osbulbul 2y ago
  yeah I am pretty sure no cloud provider thinks people who can willing to spend small money. but guess what? if one of my projects start earning money, then I will continue to what I know best :) I think that's why every cloud provider still has configs for couple of $
  - tptacek 2y ago
    We think all the time about people starting small projects here. I think a really good line to draw, if you want to understand how we think about this stuff, is between people who would be OK with us terminating their service because they mispredicted what their cap should be, and people who need their stuff to stay up and running and will just talk to support if they're concerned about billing. A shorter way to say this is that we making product pricing decisions for people who are doing this work professionally.

Poiesis 2y ago

Thomas, every time I read something you write it's a delight. I love your writing style, and it reads to me like you're always putting effort into making it succinct yet unambiguous, without unnecessary embellishment.

tptacek 2y ago
The trick is just that I write specifically for this place.

bitbasher 2y ago

There is a very obvious fix for surprise billing. Enforce a billing cap and terminate service if it's met. Even better if you send alerts when the cap is approaching. If I pay $39/month, a default cap should be $39 per-month. Otherwise, let me set a cap I am comfortable with. Surprise billing is never good for customers, only the business.

klabb3 2y ago
Yes, but I imagined that you’d have a ceiling for rolling average / token bucket over a sensible time window, with two limits: - a soft alert limit, which you set to the threshold of “hmm something is wrong but we’ll bear the cost until we figure it out” - a hard limit which fails until more tokens trickle back in, without shutting down service I really don’t want to rely on forgiveness, it’s just encouraging reckless behavior and submitting to the incomprehensibility of cloud pricing. Everyone wants these limits, why not design products with that in mind from the get go? It feels like such an afterthought
tptacek 2y ago
I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. We agree completely about surprise billing.
- sigseg1v 2y ago
  For many company's use cases (make EC2s a few times a year, otherwise leave them running) I would imagine they would really feel better if there was an option that said "if charges incurred is over (configurable threshold you set to 10k higher than your normal monthly spend), then robocall the customer, and if they do not reply after 3 automated calls then block all additional AWS API calls that would incur more than 10% of my monthly spend if left running for 24h." This would still allow all production services to run, but would stop someone from spinning up 200 crypto miners. I'm sure AWS is capable of implementing this, and I don't want to say it's "easy" but I would be shocked if they lacked the technical expertise to do this.

deleted 2y ago

[deleted]

lucifargundam 2y ago

I just wanted to say I first heard of Fly.io from "The Changelog" podcast

deleted 2y ago

[deleted]

brundolf 2y ago

I've never understood why providers don't just give you the option of setting a cost cap: "If this piece of infra exceeds [5x expected usage cost], shut it down and send me an email"

lo3p0ez 2y ago
Deno recently did: https://deno.com/blog/deploy-spend-limits https://deno.com/blog/deploy-spend-limits

veggieroll 2y ago

This thread is full of people complaining that they just want the ability to set a hard billing cap. And yet, providers continue to insist that no serious customer wants this and/or it's impossible to implement. What would it take for providers to listen to real customers here? I have $25k in cloud spend that we absolutely cannot go one single cent over due to the politics of internal budgeting. That's my reality. If you want my $25k, I need to ensure that I don't spend more than this amount. As is, my solution is to use old-school pre-rented, long-term contracted commitment VM hosting providers. This is really the only way to guarantee that you are paying an exact amount and no more. But, I really would like to use a more scalable system that didn't require pre-provisioning. And, I wish people would believe the customer when they say something and not continue to gaslight us. Providers say it's impossible, but I don't see how it would be so hard. Here's my sketch of how it could work: The main component is a system that monitors billing events and watches for the slope of the bill to ensure that there is enough runway to stay under the cap. Optionally, they could implement rate limits on resource creation to ensure that a sudden surge doesn't outrun the monitoring. You also need notifications for when the projected spend exceeds the cap. Optionally, you could implement a soft cap, where no new resources can be created. And finally, you need the hard cap where things start to get deleted. If they're feeling generous, the provider could implement a period where VM's/lambda's/etc are shut off, blob storage is not accessible, and so on so that the account holder has some time to fund the account and/or fix whatever is causing the overage. That set of features are all totally within the competency of a cloud provider. Knowing how much things cost, billing for them, and turning them on and off is their main business. And I can't believe that they expect us to believe that it's impossible to do that tracking. This is how I do system migrations. There are escalating warnings until one day the service is shutoff for an hour and turned back on. That wakes up any laggards that missed the dozen or so communications over a three-to-six month period. Finally, after a few more days, the service is shutdown for good and then data is deleted a week after that. Though I almost always keep a copy in cold storage. But that isn't necessary as a provider with a limited relationship to the client.

tptacek 2y ago
Nobody's gaslighting you. It's not impossible to build this, though it is much more difficult than it seems (cloud billing is a large-scale eventually-consistent distributed system, and if you've done any distributed systems work the issue with plugging a system like that directly into a control loop should be obvious). It's just expensive to build, and disproportionately serves the interests of customers who aren't running real apps. At $25k/mo spend, you can talk to many cloud providers (certainly including us) to work out an "I can't get invoiced for more than $25k" solution which will not involve having your app turned off abruptly when the 2,500,001st cent get spent in October. What you'll notice in this thread is that people generally want billing caps for accounts they plan to spend, like, $10/mo on. And we can build cap systems for those people --- but they'll involve turning parts of the platform off for them. I'm really pleading with people with strong opinions about hard caps to do the exercise of working out how these billing systems work. There are a huge number of apps running here, across a huge fleet of physicals, running in almost 40 regions around the world. Each of those apps has several different kinds of resources that meter at different granularities and incur different costs. Speaking as a witness to the creation of a new billing system just a month or two ago: it is kind of a miracle that these things work at all. Do the thought experiment, read some Call Me Maybe posts, and then tell me it's obvious that this feature should be straightforward to build.

theusus 2y ago

Their docs are all over the place. I don't understand what their default offerings are. And that makes it hard for me to chose them.

tptacek 2y ago
What do you mean by "default offerings"? Thanks!
- theusus 2y ago
  https://www.heroku.com/ https://www.heroku.com/ Checkout their products
  - taormina 2y ago
    Heroku is not Fly.io. Heroku was bought by the MBAs a while back, and they do something different than generic cloud hosting (they are closer to a hosted EBS). If you bother scrolling at all, you'll notice immediately below the fold that the pricing is not remotely as clear as you're pretending it is.

tptacek 2y ago

Y'all really, really want caps. We think that's a crazy feature. We don't want to do it. But I just had this conversation on a Slack with a bunch of friends at other companies and they were just as yell-y about this. There's a threshold of feedback at which I think you could get us to do some kind of cap thingo. I'm just saying. Yeesh.

ustad 2y ago
In your article or in the posts here you have not said why you think its a crazy idea.
- tptacek 2y ago
  I've said it over and over again: if you have a serious app, and someone somehow steals a credential from you and uses it to light up a bunch of crypto miners, you don't want us shutting down your main app in response. We perceive it mostly as a feature that will blow people up. We agree about the underlying problem! You don't want to spend $5000 in a month for services you never wanted. We don't want you spending that either. We'd rather just improve our billing so that you can fix this after the fact without trading off availability. But I have just planted a flag: we are prepared to be wrong about this. I don't think we are, but like, I'm the only one. :)
  - ustad 2y ago
    Crypto miners is the reason for not having a “crazy” option for “cap on/off” at sign up? You know what you can’t fix after the fact of getting a huge bill? a heart attack.
    - tptacek 2y ago
      This post is about us not giving you that heart attack. I'm pushing back, and I'm going to keep pushing back; if we do this feature, it is going to be kicking and screaming. :)
  - bee_rider 2y ago

knowitnone 2y ago

no matter what, you'll have some fraud. Perhaps a better solution is the customer has a know they can turn that sets max limits and some alerts on when usage is abnormal.

csours 2y ago

> " You probably can’t tell me how much electricity your home is using right now, and may only come within tens of dollars of accurately predicting your water bill. But neither of those bills are all that scary, because you assume there’s a limit to how much you could run them up in a single billing interval. " I had a $600 surprise water bill. It was (partially) forgiven because the water department could drive to my house and see evidence of the leak next to my water meter. It did turn out to be on my side of the meter, so it is my responsibility. If the water department had driven to my house and seen evidence of commercial agriculture (so to speak), then it would not have been forgiven. --- The parallel here is that the water department can't come into my house uninvited - the cloud provider SHOULD NOT have intimate access to the running code, but they are able to observe some patterns without 'breaking in'. --- Side note: the size of the bill and the amount of forgiveness was largely driven by waiving an 'excess usage' surcharge - similar to how you can get a discount for cloud service reservations.

wrs 2y ago
Water billing is indeed nonlinear with no cap. I had a surprise $3000 excess water bill due to a broken connection at the meter. Followed by a $3000 excess sewer bill (!) because the assumption is if I used $3000 of water, it must have gone down the drain. However, if you demonstrate to an inspector that the water must have leaked below ground (as in this case) there’s a process for getting the second charge waived. Unfortunately the leak was on our side of the meter, so the first charge was correct, though we got partial forgiveness.
- genewitch 2y ago
  So, when they read the meter, and saw that it was flooded at the meter or nearby, they didn't think to mention that to you? I ask because you said "surprise bill" - another comment i made explains my neighbor's water meters have a light on them that lights up if there's a leak, which the meter reader can see and the homeowner usually cannot - the only reason to not notify the homeowner of an issue is because of $.
  - 2y ago

ctennis1 2y ago

Google Cloud will not cut you a break unless you have an incredible about of pull. Even then, you won't get much of one.

smithcoin 2y ago
I'm suffering for this now. They just erroneously charged me almost $4,000 out of nowhere because they decided to scan all of old container images for no reason and without notification. I have a support partner and everybody is just passing the buck around, it's super frustrating.
aidenn0 2y ago
Other comments here (e.g. https://news.ycombinator.com/item?id=41333600 https://news.ycombinator.com/item?id=41333600) suggest that this is not universally true.
campers 2y ago
Being at the level where you have an account manager should help. Our technical account manager always said if we ever accidentally rack up a bunch of cloud costs we should at always ask if we can get a refund, and as long as its infrequent there's a good chance to get some credits.

spencerchubb 2y ago

When I was in highschool, my friends and I were doing a small project. My friend accidentally ran Google's image recognition service in a while loop for a whole minute, and racked up a $300 bill. Thankfully GCP waived it for us.

reaperman 2y ago
I ran up a $4,000 bill with like a week of TPU time bc I didn’t realize that they kept running after the attached CPU turned off (unlike GPU offerings). GCP also waived the whole thing - even refunding the $4k they had already pulled from my payment method.
- spencerchubb 2y ago
  That's brutal, and if you're anything like me, probably learned a lesson very thoroughly for the future!

ultra_nick 2y ago

Build everything on Docker, Kubernetes, and VPS. Run on the cheapest VPS provider with human support. Make a quick and easy switch if they get too shifty.

andrewstuart 2y ago

Just run Linux computers on ionos with unlimited traffic and get on with the job instead of worrying about silly things like how much traffic passed over some internal software system and will it bankrupt the company. https://www.ionos.com/servers/vps https://www.ionos.com/servers/vps

ilrwbwrkhv 2y ago
Same but on Hetzner. Incredibly beefy machines. Unlimited traffic. No fear of any sudden changes that I explicitly didn't act upon. I do not know why people fall for the shiny marketing so easily, that too hackers? Is it all the Javascript folks? Vercel I know is propped up by them for sure.
- tptacek 2y ago
  I am never going to get tired of telling people that if they can solve their problems running on Hetzner, they should do that. It's not like a dirty secret of our industry. By all means, deploy with VPSs! Or on bare metal servers!
  - ilrwbwrkhv 2y ago
    Do you have a blog post on fly's blog where you tell people that?
    - tptacek 2y ago
      We are incredibly backlogged on the blog. I've got 3000 words on Tigris rolling out next week, we've got the results of a $300K 3rd party GPU audit to write up, we wrote a replacement to Vault called Pet Semetary, we need to write up the billing system because, as you can see from this thread, people appear to think that's not a hard technical problem. I can imagine getting around to writing a post about when a VPS is a good alternative to launching a container on Fly.io, but it'll be aspirational; I doubt I'll ever get around to it. But I'm happy to remind people here of it!
    - 2y ago

aidenn0 2y ago

IANAF, but Lower Montana seems like a terrible place to farm coffee beans?

maverwa 2y ago
So Kurt is going to hate that job, too?
tptacek 2y ago
I asked him about this and he didn't explain, but did demand that I keep it. He has a weird thing about Montana.

Aeolun 2y ago

> may only come within tens of dollars of accurately predicting your water bill Uh, what is the magnitude of these water bills? I’m fairly certain I can predict it to within a few dollars because it’s only $35 in the first place.

lupire 2y ago
My water bill is $200/quarter.
School-Cotton 2y ago
I have absolutely no idea how much I pay for water, but vaguely think it’s probably between 0 and 100. Why does knowing it’s small mean you can predict it precisely, in a world where auto-payment exists so you never have to actually look at your bills?
- Aeolun 2y ago
  I mean that it’d be extremely unlikely for me to not be able to predict my water bill to within ‘tens of dollars’, because that is the entire amount. Also, I get receipts by postal mail even if I auto pay them, so I guess that makes it easier.

srazzaque 2y ago

Whilst it doesn't have the mindshare or some features which are tablestakes for enterprise customers, I've found Linode's pricing to be extremely predictable for small projects. Even post Akamai acquisition. I'm sure other smaller players are also just fine. Couple that with the fact you can achieve quite a bit on simple set-ups that are adequately sized to begin with, you can save quite a bit. Not all of us need elasticity, or environments being spun up/down on commit.

tonymet 2y ago

It's important that all utility metering and pricing is effectively made up to balance costs across customers. That's why residents in california probably pay 100000x / unit of water than agriculture customers do . The same works for the cloud. An operation that costs $100000 does not cost the operator $85000 in operations on the margins. Yes the metering and pricing schedules are necessary, but punitive pricing for accidents really is just an artifact of the system. I'm guessing the biggest reason AWS provides forgiveness is that if anyone took them to court over a bill, a court would throw out the charge once the wholesale cost was revealed.

kasey_junk 2y ago

If I were fly I’d implement spending caps just to filter out unserious operators. My implementation of that feature would be when you turned it on fly would just kick you off. No serious operator wants a provider to turn their system off. So if you want that it’s pretty clear you are hosting a silly system. Which likely costs more to support and drives less margin.

_jab 2y ago
No serious operator would choose a provider with your implementation of that feature.
- tptacek 2y ago
  I agree that no serious operator would choose a provider whose spending cap feature was "enable this feature and we kick you off the platform".
  - psd1 2y ago
    People will pay to be abused: c.f. Oracle
lo3p0ez 2y ago
Hu? For any business having a due to a cap throttled service is better than losing the money straight away due to an accidental cloud bill explosion.
icedchai 2y ago
The real solution is to make it simpler to set up billing alerts. If my hobby project is ever projected to cost more than $0.50/hr, let me know. Billing alerts should be part of onboarding for all cloud providers, especially for individuals that don’t want surprises.
langcss 2y ago
And yet every single cloud let's you scale from zero and play and experiment. They know today's tinkerers are tomorrow's CTO decision makers.

lo3p0ez 2y ago

[flagged]

jowdones 2y ago

That's why my phone SIM is still not a subscription but rather I manually buy credit each month. The price is even lower than subscription and benefits exceed it, shitloads of Gb and minutes. If I'll use roaming I'll pay for a more expensive credit that month but I have piece of mind I won't be charged thousands of euro at the end of the month. Like telecom providers of course cloud providers could have metered service billed under "get only as much as you paid" policy but it's obviously they totally and in bad faith don't want that.

figassis 2y ago

This is why I love Hetzner. It has billing alerts. What was so difficult about it that fly had to rebuild their system? Also, do everything except databases inside kubernetes. Deploy kubernetes across multiple clouds via wireguard. Label your instances properly on each provider. Prefer bare metal instances where available. Migrate your workloads accordingly. Force cloud providers to earn your money. Don’t however have both workloads running at the same time in multiple providers as you will eat insane data costs.

psd1 2y ago
I can't read the tone, but perhaps you haven't seen much technical debt. It's possible to have a fairly good codebase, full of good decisions, yet you still can't "get there from here". Second paragraph was useful, thanks!

campers 2y ago

> Meanwhile: like every public cloud, we provision our own hardware, and we have excess capacity. Your messed-up CI/CD jobs didn’t really cost us anything This^ Don't feel bad about asking for credits when you accidentally make a costly mistake.

nottorp 2y ago

Isn't it easier to set hard limits on the account and notify the client when they're close to them so they can decide to increase them on their own if needed? But then you run into the risk that the client will never increase those hard limits and ... pay less than they could. Not good for business.

bruce343434 2y ago

So AI is now being used on the Cloud to fight the Blockchain?

deleted 2y ago

[deleted]

lolinder 2y ago

> The Fully Automated Accident-Forgiving Billing System of the Future (which we are in fact building and may even one day ship) will give you a line-item veto on your invoice. For a lot of the personal projects and early stage startups that are most terrified of these kinds of mistakes and therefore avoid autoscaling products like fly.io, we sincerely would rather have the entire account shut down when it goes over budget than ever see a bill that's higher than our net worth. A line item veto somewhat alleviates that concern but not fully. They indicate at the end that they're going to do something along these lines, but what they're describing there also seems over-engineered compared to a simple circuit breaker that kills the account or some subset of it. Is there a good reason for these providers to avoid implementing that feature, which on a naive look seems far simpler than what they've actually proposed?

tptacek 2y ago
Something like half the comments on this story are a discussion of why or why not cloud providers do or don't provide this simple circuit breaker feature.
- lolinder 2y ago
  I read yours after posting this and it provides exactly zero information: > I promise, you are not the first person to have thought of this, and, believe it or not, there are reasons other than malice and avarice that cloud providers don't terminate service based on billing caps. Terminating service is a big deal. "Terminating service is a big deal" how? I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend? No one is asking for automatic circuit breakers applied to all customers indiscriminately, but I'm not seeing any justification in here for why an opt-in circuit breaker is technically or legally challenging to implement.
  - kikoreis 2y ago
    > I can explicitly cancel a subscription after a certain date—what is the problem with me explicitly cancelling a subscription after a certain amount of spend? As someone also involved in billing systems for public clouds: in theory there's no difference, but in practice there is a world of difference. This is the sort of situation where the end user is commonly surprised with the consequences of their own decisions. At MGC we have some "soft shut down" processes, and we constantly hear stuff like "I know I said shut down, but this is the one situation where that really didn't make sense"; where examples are "storage which keeps backups became unavailable", "a very simple but critical user auth system disappeared", "I had no idea this was still running on my account", or "OMG not in the middle of the weekend", etc. You can build heuristics and tracking into the system to minimize these situations, but that's a lot of work. So yeah, it is a valid use case and something many CSPs would like to provide, but implementing something that is actually better than nothing is non-trivial.

timoxley 2y ago

I feel like this indicates there's a market for traffic-spike insurance