54 ms·

Serverless Horrors

626 points by operator-name 1y ago

trcf22 1y ago
After a quick check on Vercel stories, it seems all payments were discarded or mistakes in the first place. Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?
- Alifatisk 1y ago
  > Do you need to tweet about it to be reimbursed? This is what scares me, is social media the only way to get things sorted out nowadays? What if I don't have a large following nor an account in the first place, do I have to stomach the bill?
  - wg002 1y ago
    I can't imagine them sending it to collections. What kind of recourse would a company like Vercel have if you don't pay it?
    - immibis 1y ago
      They can sue you for the bill plus the legal costs.
  - pelagicAustral 1y ago
    This is exactly what happened to me during Covid... I had a flight that got cancelled at the beginning of the pandemic since the country closed the orders (essentially). A year after, still on lock downs and et al, I wanted to enquire about a refund, for months I got not answer, until I caught wind that people using Twitter were actually getting results. Now, I don’t use social media at all, so I had to create a Twitter account, twit about my case et voila! 30 mins after I got a response and they send me a PM with a case number... Not even going to mention the airline, but it is infuriating...
- pjmlp 1y ago
  No, at least in enterprise consulting for these kind of hosting, usually there is a contact person on the support team that one can reach directly. However these projects are measured in ways that make Oracle licenses rounding errors. Which naturally creates market segmentation on who gets tier 1 treatment and everyone else.
  - viraptor 1y ago
    Once you're in a contract + TAM territory, pricing works very differently. Also, temporary experiments and usage overruns become an interesting experience where the company may just forget to bill you a few thousands $ just because nobody looked at the setup recently. Very different situation to a retail user getting unexpected extra usage.
- deleted 1y ago
  [deleted]
- Havoc 1y ago
  Relying on the mercy of a support agent that may be having a bad day is a poor strategy
- tonyhart7 1y ago
  I mean if developer got charged with 100k, more often than not the bank would decline that first maybe if you didn't have that high credit limit but what happen if this happen to corporate account and somewhere resource get leaked??? multi billions dollar company probably just shrug it off as opex and call it a day
- interloxia 1y ago
  Someone at a community group I'm in messed up playing with Azure through their free for non-profits offering^. We were out about 1.2k€. Not huge but huge for us. Encouraged by comments on HN over the years I had them ask support to kindly to wave it. After repeating the request a few times they eventually reduced their bill to <100€ but refused to wave it entirely. So even without shaming on social media, But it probably depends. It's worth at least asking. ^The deal changed about six months ago.
  - cuu508 1y ago
    It's waive, not wave
joshstrange 1y ago
I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one: https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c... About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.
- siva7 1y ago
  How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
- sherburt3 1y ago
  I believe they changed that shortly after that blog post went viral: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
  - franktankbank 1y ago
    Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
    - pooper 1y ago
      > Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great! It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior". > AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception. The stench of this bovine excrement is so strong that it transcends space time somehow.
wg002 1y ago
This site is a bit dated. I remember in response to this Vercel added a way to pause your projects when hitting a spend limit. I enabled it for my account. Still, it made me question why I'm not using a VPS.
- aurareturn 1y ago
  Vercel used to be called Zeit. They had a server product called Now that gave you 10 1CPU/1GPU instances for $10/month (or $20 I forgot). It was the best deal. When Vercel switched everything to serverless, it all became pretty terrible. You need 3rd party services for simple things like DB connection pooling, websockets, cron jobs, simple queue, etc because those things aren’t compatible with serverless. Not to mention cold starts. Just a few weeks ago, I tried to build an API on Next.js+Vercel and get random timeouts due to cold start issues. Vercel made it easier to build and deploy static websites. But really, why are you using Next.js for static websites? Wordpress works fine. Anything works fine. Serverless makes it drastically harder to build a full app with a back end.
  - pjmlp 1y ago
    Serverless is the most common deployment on MACH projects. Because when everything is a bunch of SaaS Lego bricks, serverless is all one needs for integration logic, and some backend like logic. Add to it that many SaaS vendors in CMS and ecommerce space have special partner deals with Vercel and Nelify. https://macharchitecture.com/ https://macharchitecture.com/
dakiol 1y ago
> I had cloudflare in front of my stuff. Hacker found an uncached object and hit it 100M+ times. I stopped that and then they found my origin bucket and hit that directly. Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?
- solatic 1y ago
  No. Your buckets should be private, with a security rule that they can only be accessed by your CDN provider, precisely to force the CDN to be used.
  - rwmj 1y ago
    Why isn't that the default? I'm glad I use a Hetzner VPS. I pay about EUR 5 monthly, and never have to worry about unexpected bills.
    - graemep 1y ago
      Because not all uses for buckets fit that. Buckets are used for backups, user uploads, and lots of things other than distributing files publicly.
      - graemep 1y ago
        I would say its probably not a good idea to make a bucket directly publicly accessible, but people do not do that. A lot of the point of serverless is convenience and less admin and things like adding a layer in front of the bucket that could authenticate, rate limit etc. is not convenient and requires more admin.
    - kdps 1y ago
      Don't they charge for every TB exceeding the included limit? (website says "For each additional TB, we charge € 1.19 in the EU and US, and € 8.81 in Singapore.")
the__alchemist 1y ago
"Serverless" is a an Orwellian name for a server-based system!
- magnusm 1y ago
  Thats true!
- Biganon 1y ago
  "There's no cloud; it's just someone else's computer"
- Spivak 1y ago
  But your so called "no-code" system runs on code. Checkmate atheists. There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
  - StevenWaterman 1y ago
    and your wireless modem has wires
  - zahlman 1y ago
    In the "no-code" system, the end user does not write code. In the "serverless" system, the end user does connect to a server. It doesn't just "have" servers; they aren't a hidden implementation detail. Connecting to a website is an instrumental part of using the software.
    - viccis 1y ago
      "Serverless" refers to the demarcation point in the shared responsibility model. It means there aren't any servers about as much as "cloud hosting" means the data centers are flying.
    - immibis 1y ago
      In the "code" system the end user does not write code either - that's the developer's job. In the "no code" system it's the developer who doesn't write code, and in the "serverless" system it's the developer who doesn't set up servers.
zkmon 1y ago
Maintaining your own containers or VMs is hard considering how much risk appetite you have for the issues at infra level. So, yeah, when you complain about the costs of serverless, you are just paying for your low risk appetite low cost of your IT management.
paseante 1y ago
Yeah I also left my website hosted on Google Cloud because costs popped from everywhere, and there is basically no built-in functionality to limit them. So I didn't really slept relaxed (I actually slept great, but I hope you get the point) knowing that a bug could cost me... who knows how much. Actually, as the website of OP says, for spending control you have budget notifications and with that you can disable the billing for all the project altogether through some API call or something, I don't remember exactly, that is all there is. But still it looks like this functionality is just not there.
- mortsnort 1y ago
  You can write Google cloud functions to disable your credit card when certain thresholds are met pretty easily, but it's unethical that this isn't just a toggle somewhere in settings.
  - Cpoll 1y ago
    Does that actually stop the spend immediately? If not, you're still on the hook for the bill. I suppose you can walk away and let them try to come after you, but that wouldn't work for a company.
EGreg 1y ago
Are there any protections these days at the cloud provider level? Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?
- bc569a80a344f9c 1y ago
  Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes. That means there’s some lag, and you are responsible for any costs incurred that go over budget between such reporting runs.
  - chuckadams 1y ago
    > of course AWS doesn’t check each S3 request or Lambda invocation against your budget If it can bill them per-invocation, why can't it also check against a budget? I don't expect it to be synchronous, but a lag of minutes to respond is still better than nothing. Can you even opt-in to shutting down services from the budget tool, or is that still something you have to script by hand from Cloudwatch alarms?
    - bc569a80a344f9c 1y ago
      You script it by hand. I think figuring out how to do this faster is less trivial than it might sound. I agree that synchronous checks aren’t reasonable. But let’s take Lambdas. They can run for 15 minutes, and if you consolidate within five minutes after a resource has been billed, that gives you a twenty minute lag. I’m not trying to make apologies for Amazon, mind you. Just saying that this isn’t exactly easy at scale, either. Sure, they bill by invocation, but that’s far from synchronous, too. In fact, getting alerts might very well be happening at the frequency of billing reconciliation, which might be an entirely reasonable thing to do. You could then argue that that process should happen more frequently, at Amazon’s cost.
Havoc 1y ago
Putting any sort of pay per use product onto the open internet has always struck me as insane. Especially with scaling enabled. At least stick a rate limited product in front of it to control the bleed. (And check whether the rate limit product is in itself pay per use...GCP looking at you)
deleted 1y ago
[deleted]
omnicognate 1y ago
It would help to round to the cent. With 3 digits to the right of the dot it's ambiguous whether it's a decimal point or a thousands separator, and the font and underline makes the comma vs dot distinction a bit unclear.
- thedanbob 1y ago
  A number of the titles appear to have 69 or 420 cents added to the amount that appears in the story.
jamil7 1y ago
Don’t most of these services have config options to protect against doing this? I haven’t used most of these services but it running up a bill during traffic spikes but not going down seems like it’s working as intended?
- swiftcoder 1y ago
  Nope, basically none of these services have a way to set a hard budget. They let you configure budget warnings, but it’s generally up to you to login and actually shut down everything to prevent from being billed for overages (or you have to build your own automation - but the billing alerts may not be reliable)
- hxtk 1y ago
  I know AWS in particular does not because they do not increment the bill for every request. I don't know exactly how they calculate billing, but based on what I do know about it, I imagine it as a MapReduce job that runs on Lambda logs every so often to calculate what to bill each user for the preceding time interval. That billing strategy makes it impossible to prevent cost overruns because by the time the system knows your account exceeded the budget you set, the system has already given out $20k worth of gigabyte-seconds of RAM to serve requests. I think most other serverless providers work the same way. In practice, you would prevent such high traffic spikes with rate limiting in your AWS API Gateway or equivalent to limit the amount of cost you could accumulate in the time it takes you to receive a notification and decide on a course of action.
ChrisMarshallNY 1y ago
At one time, I considered using Firebase as a backend, but then, I kept reading stories like these, and decided to roll my own. I'm fortunate to be able to do that. It's kind of amazing, though. I keep getting pressure from the non-techs in my organization to "Migrate to the Cloud." When I ask "Why?" -crickets. Industry jargon has a lot of power. Seems to suck the juice right out of people's brains (and the money right out of their wallets).
phoenixhaber 1y ago
When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money. I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.
- croes 1y ago
  That’s why I prefer prepaid cards or those I can easily freeze to prevent any booking.
  - jsheard 1y ago
    If your card is declined and they don't feel like forgiving the bill, won't they just send debt collectors after you instead?
    - croes 1y ago
      Yes, but it's better they need to get their money than you need to get your money back. 100.000 easily can put you in ruining debt. It's the better position to still have your money even if you have to pay.
  - kleinsch 1y ago
    Freezing a card doesn’t mean the debt is erased. They can still take you to collections.
nurettin 1y ago
I've had this twice. Once with oracle, once with azure. They both charged me $2000-$5000 for simply opening and closing a database instance (used only for a single day to test a friend's open source project) To be fair, support was excellent both times and they waived the bills after I explained the situation.
- qcnguy 1y ago
  How did you run up a $5000 bill for just testing a project? What kind of project was it that could put so much load on the DB?
  - deleted 1y ago
    [deleted]
caboteria 1y ago
The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right? I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.
- hvb2 1y ago
  You don't think this happens on prem? Servers running an application that is no longer used? Sure they're probably VMs but their cost isn't 0 either
  - sgarland 1y ago
    With that model, your cost doesn't change, though. When/if you find you need more resources, you can (if you haven't been doing so) audit existing applications to clear out cruft before you purchase more hardware.
    - remus 1y ago
      That's the equivalent of saying "just audit your cloud usage and remove stuff that's no longer used".
    - hvb2 1y ago
      The cost of going through that list often outweighs the cost of the hardware, by a lot. And in a lot of cases it's hard to find out if a production application can be switched off. Since the cost is typically small for an unused application, I don't know if there are many people willing to risk being wrong
      - sgarland 1y ago
fnord77 1y ago
looking forward to the "LLM token horrors" version
- mdaniel 1y ago
  I thought there was an OWASP for "denial of wallet" vulnerabilities but this link was the closest one I found https://www.prompt.security/vulnerabilities/denial-of-wallet-service https://www.prompt.security/vulnerabilities/denial-of-wallet... (although the link makes it sound like they're offering denials)
  - kjs3 1y ago
    We call them "Billing Fatigue Attacks". Keep an eye on your autoscaling, kids...
api 1y ago
There should also be a general category for "cloud horrors" for things that cost $50k/month to host that would be $1500/month on a bare metal provider like Datapacket or Hetzner. I'm old enough to remember when cloud was pitched as a big cost saving move. I knew it was bullshit then. Told you so.
- nchmy 1y ago
  even $1500/mo on hetzner is a seriously large app. You could get 300 cpus and 1.5TB of RAM for that price.
emseetech 1y ago
This is why when I contract for an early stage startup, I pose the question: "What if your app went viral and you woke to a $20k cloud bill? $50k? $80k?" If the answer is anything less than "Hell yeah, we'll throw it on a credit card and hit up investors with a growth chart" then I suggest a basic vps setup with a fixed cost that simply stops responding instead. There is such a thing as getting killed by success and while it's possible to negotiate with AWS or Google to reduce a surprise bill, there's no guarantee and it's a lot to throw on a startup's already overwhelming plate. The cloud made scaling easier in ways, but a simple vps is so wildly overpowered compared to 15 years ago, a lot of startups can go far with a handful of digitalocean droplets.
mococa 1y ago
Couple years ago I was charged in USD 4K on Google Cloud after trying recursive cloud functions. I told them that was a mistake and they forgot the debit, they just asked to no do again.
game_the0ry 1y ago
I have a feeling I will be downvoted for this, but... Have the people posting these horror stories never heard of billing alerts?
- skippyboxedhero 1y ago
  Many of the stories on the site are from people who have billing alerts. If you have bot spam, how do you actually think their billing alerts work? The alert is updated every 100ms and shuts off your server immediately? That isn't how billing alerts can or should work.
  - McGlockenshire 1y ago
    Yes, actually, if continuing to run the service is going to exceed my available budget then I do want the service turned off! If I can't pay for it, and I know I can't pay for it, what other possible choice do I have? Do any of you people have budgets, or do you all rely on the unending flow of VC money?
    - skippyboxedhero 1y ago
      That isn't how this can work. If you are running a service and then find out that AWS is spamming you every 100ms to find out what your CPU is doing (or calling out every 100ms) then people would be quite unhappy. The majority of these massive bills are due to traffic, there is pretty much no way that AWS could stop your server in time...if they had the choice, which they don't. I think my original point was unclear: I am pointing out that if you just think about how this stuff can possibly work, billing alerts can not work in the way you expect. The alert is updated async, the horse has bolted and you are trying to shut the gate. I don't use AWS for personal stuff because I know their billing alerts won't stop me spending a lot. Don't use them if that is a concern. I do use AWS at work, we are a relatively big customer and it is still very expensive for what it is. The actual hardware is wildly overpriced, their services aren't particularly scalable (for us), and you are basically paying all that overage for network...which isn't completely faultless either. Imo, using them in a personal capacity is a poor idea.
johnebgd 1y ago
We are building bare metal for our workloads… I don’t care if cloud is supposed to be cheaper because it never is. You can get a decent small business firewall to handle 10gbit fiber for $600 from unifi these days. Just another reason I’m glad I moved out of the Bay Area and nyc to a midwestern town for my company. I have a basement and can do rad things in my house to grow my business.
- nchmy 1y ago
  bUt wuT aBowT deV OpS?!
1oooqooq 1y ago
last employer asked for an estimate to migrate to cloud. it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr. ceo told to go ahead anyway (turn out company was being sold to Apollo) first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one. can't imagine how companies miss that step.
skippyboxedhero 1y ago
Hetzner, 16TBx2 HDD, 1TBx2 SDD, 64GB RAM, 20TB free bandwidth, $70/month. I used 1TB of traffic on a micro instance and it cost me $150 (iirc). Doesn't have to be this way.
shayway 1y ago
I guess I'm missing something, why is this 'serverless' horrors? If anything it seems to specifically be serverful horrors.
- o11c 1y ago
  "Serverless" is just marketing-speak for "somebody else's server".
stressback 1y ago
I read a lot of the posts at the little blog here and, uh, every single one sounds like a complete amateur making a cloud configuration mistake. I haven't found one that is the provider's fault or the fault of "serverless" I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself. Serverless isn't a horror, serverlesshorrors poster. You are the horror. You suck at architecting efficient & secure systems using this technology, you suck at handling cloud spend, and you suck at taking responsibility when your "bug" causes a 10,000x discrepancy between your expected cost and your actual bill. Just because you don't understand it doesn't mean it sucks
- Cpoll 1y ago
  You're not wrong about cloud configuration mistakes, but a tool that lets you increase costs 10000x (without even letting you set a safety) is a hell of a chainsaw. I'm more worried about the overconfident SRE that doesn't stay up at night worrying about these.
- Bluecobra 1y ago
  That being said, the cloud providers could do a better job explaining to new/naive users that great power comes with great responsibility and there is no hand holding. Someone might be more hesitant to willy nilly spin up something if a wizard estimates that the maximum cost could be $X per month.
- webdevver 1y ago
  truth nuke
- McGlockenshire 1y ago
  > every single one sounds like a complete amateur making a cloud configuration mistake Golly if only the configuration wasn't made this way on purpose exactly to cause this exact problem.
- bhk 1y ago
  Consider this analogy: Instead of using a root command shell, it is wise to use an account with appropriately restricted capabilities, to limit the downsides of mistakes. Cloud services support the notion of access control, but not the notion of network resource usage limits. It's an architectural flaw. Or do you always log in as root, like a real man, relying purely on your experience and competence to avoid fat-finger mistakes?
luxuryballs 1y ago
there should be some kind of insurance for bugs that introduce unusually expensive usage
- mdaniel 1y ago
  I believe any such policy would need its premiums based on the services used (and likely the qualifications of the staff) since, unlike rebuilding a house, the financial risk is almost unlimited with out of control cloud spend It reminds me of the Citi(?) employee who typed the wrong decimal place in a trade: computers make everything so easy!
deleted 1y ago
[deleted]
NullCascade 1y ago
Troy Hunt and HIBP is a good example in the other direction but Hunt has also been burned plenty of times by serverless. https://www.troyhunt.com/closer-to-the-edge-hyperscaling-have-i-been-pwned-with-cloudflare-workers-and-caching/ https://www.troyhunt.com/closer-to-the-edge-hyperscaling-hav...
dinvlad 1y ago
Andras (author of Serverless Horrors) knows what he’s talking about. The amount of brainwashing that big cloud providers have done, is insane.
petralithic 1y ago
This is some good marketing for Coolify, which the author makes as an open source platform as a service. I prefer Dokploy these days though, since it seems to be less buggy, as Coolify seems to have such bugs due to being on PHP. https://coolify.io/ https://coolify.io/ https://dokploy.com/ https://dokploy.com/
- khromov 1y ago
  CapRover is another good alternative, and also much more lightweight than Coolify, easily runs on even a 512MB server: https://caprover.com/ https://caprover.com/
andrasbacsai 1y ago
look mom & dad, I am famous!
adamddev1 1y ago
I remember at the beginning of the serverless hype how they said it was great because it automatically scaled as big as you need it. Given how sudden and massive these "scaling spikes" can be, I would much rather deal with a death-hugged VPS than a $100k bill. Plus the VPS is just so much faster in most cases.
acoustics 1y ago
The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent. Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build. I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend? Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?
- rr808 1y ago
  Maybe, but its a huge reason to use real servers instead of serverless.
  - pixl97 1y ago
    I mean real servers get hit with things like bandwidth fees so it's not a 100% solution.
    - estimator7292 1y ago
      Not even remotely the same scale of problem. Like at all. If your business suddenly starts generating Tbs of traffic (that is not a ddos), you'd be thrilled to pay overage fees because your business just took off. You don't usually get $10k bandwidth fees because your misconfigured service consumes too much CPU. And besides that, for most of these cases, a small business can host on-prem with zero bandwidth fees of any type, ever. If you can get by with a gigabit uplink, you have nothing to worry about. And if you're at the scale where AWS overages are a real problem, you almost certainly don't need more than you can get with a surplus server and a regular business grade fiber link. This is very much not an all-or-nothing situation. There is a vast segment of industry that absolutely does not need anything more than a server in a closet wired to the internet connection your office already has. My last job paid $100/mo for an AWS instance to host a GitLab server for a team of 20. We could have gotten by with a junk laptop shoved in a corner and got the exact same performance and experience. It once borked itself after an update and railed the CPU for a week, which cost us a bunch of money. Would never have been an issue on-prem. Even if we got DDoSed or somehow stuck saturating the uplink, our added cost would be zero. Hell, the building was even solar powered, so we wouldn't have even paid for the extra 40W of power or the air conditioning.
princevegeta89 1y ago
In my experience: Fuck serverless. If we're building anything bigger than a random script that does a small unit of work, never go for serverless. A company I recently worked for went with Serverless claiming that it would be less maintenance and overhead. It absolutely was the worst thing I've ever seen at work. Our application state belonged at different places, we had to deal with many workarounds for simple things like error monitoring, logging, caching etc. Since there was no specific instance running our production code there was no visibility into our actual app configuration in production as well. Small and trivial things that you do in a minute in a platform like Ruby on Rails or Django would take hours if not days to achieve within this so-called blistering serverless setup. On top of it, we had to go with DB providers like NeonDb and suffer from a massive latency. Add cold starts on top of this and the entire thing was a massive shitshow. Our idiot of a PM kept insisting that we keep serverless despite having all these problems. It was so painful and stupid overall.
- kikki 1y ago
  Why was your PM making tech decisions?
  - Nextgrid 1y ago
    Looks like you need the "quiet part" said out loud: Chances are, the company was fishing for (or at least wouldn't mind) VC investment, which requires things being built a certain (complex and expensive) way like the top "startups" that recently got lots of VC funding. Chances are, the company wanted an invite to a cloud provider's conference so they could brag about their (self-inflicted) problems and attract visibility (potentially translates to investment - see previous point). Chances are, a lot of their engineering staff wanted certain resume points to potentially be able to work at such startups in the future. Chances are, the company wanted some stories about how they're modern and "cloud-native" and how they're solving complex (self-inflicted) problems so they can post it on their engineering blog to attract talent (see previous point). And so on.
    - princevegeta89 1y ago
zahlman 1y ago
I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work. To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an Internet connection, it's not to send data to a specific place, under the developer's control, for the purpose of making the software system function as advertised.
- fritzo 1y ago
  It's like a company with no employees. There are still people performing services, but on temporary contracts.
- maxbond 1y ago
  That's generally called "local." Serverless is poorly named but describes how certain backends are deployed, not applications without a backend.
- leptons 1y ago
  A "Server" is typically a single machine that has a specific OS and runs layers of various software that allows your business logic to be accessed by other computers (by your users). For a "Server" you typically have to choose an OS to run, install all the support software (server monitoring, etc), update the software, and if the server fails you have to fix it or rebuild it. With "Serverless", your code is in a "function as a service" model where all you have to worry about is the business logic (your code). You don't have to set up the server, you don't have to install the server OS, or any basic server software that is needed to support the business logic code (http server, etc). You don't have to update the server or the underlying server software. You don't have to perform any maintenance to keep the server running smoothly. You never (typically) have to worry about your server going down. All you have to do is upload your business logic function "somewhere" and then your code runs when called. Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic. That's why it's called "Serverless" because you don't have to deal with any of the hassle that comes with running an actual "server".
jppope 1y ago
This is a weird take on an incredibly useful paradigm (serverless). One the one side, there are obviously precautions that all of these users could have taken to avoid these charges on the other hand its totally common to spin up a thing and forget about it or not do your due diligence. I totally feel for the people who have been hit with these chargers. At the end of the day though the whole think feels like a carpenter shooting themselves in the foot with a nail gun then insisting that hammers are the only way to do things.
mahirsaid 1y ago
Seem likes there are mistakes that were made on behalf of the users. The attackers found these mistakes and took advantage of them. i don't think "severless" is the problem.
- bapak 1y ago
  Serverless is the problem in that most serverless services don't let you hard-cap spend. This issue is serverless-specific. If I pay $20/month on VPN the most frightening thing that can happen is the client calling you about your website being down, not a $100k bill.
general1465 1y ago
I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything. That's just a problem waiting to happen while you are always running tests on production...
- scarface_74 1y ago
  This is nowhere near being true.
- nostrebored 1y ago
  1. Put your stuff in a stack. Deploy it to your isolated developer account. Basically free staging environment. 2. Use the officially supported docker runtime for local testing. 3. Treat it like any other code and make unit tests 4. Use one of the tools like localstack to emulate your staging env on your machine. There are so many options that I don’t know how you could walk away with your opinion.
  - Nextgrid 1y ago
    Or you could just write conventional software. But I get it, you don't get resume points nor invites to cloud-provider conferences for that. > Basically free staging environment. [emphasis mine] Not really. Sure, the cost would usually be peanuts... until you have an infinite loop that recursively calls more lambdas. Then you have a huge bill (but hey that pays for your invites to their conferences, so maybe it's a blessing in disguise?). And yes, you will pretty much always get it refunded, but it's still a hassle and something that is absolutely not necessary. Snark aside, having an opaque dev environment always constrained by bandwidth and latency that can’t be trivially backed up/duplicated is a terrible idea and why I always recommend against “serverless”, even besides the cost concerns. Serverless is OK for small, fully self contained pieces or code that are fire and forget. But for anything complex that’s likely to require maintenance, no thanks.
  - rurp 1y ago
    Eh, I worked on a large serverless project that worked hard to follow best practices but it was still very clunky to run and test code locally. The local serverless tools simply didn't work for our project and they had so many limitations I'm skeptical they work for most non-prottypes. Deploying a stack to your own developer environment works fine and is well worth doing, but the turnaround time is still painful compared to running a normal web framework project locally. Deploying a stack takes much much longer than restarting a local server. Serverless isn't all bad, it has some nice advantages for scaling a project, but running and debugging a project locally is a definite weak spot.
CivBase 1y ago
I can't imagine hosting a small-time project on rented infrastructure without some kind of mechanism to terminate it once costs exceed a reasonable threshold.
mitjam 1y ago
I once found an official Microsoft example repo to deploy an LLM gateway on Azure with ALB. Glad I did the tedious work of estimating the costs before I hit the deploy button (had to go though many Biceps manifests for that). The setup would have cost me about 10k/month.
nromiun 1y ago
I was also too careless with AWS when I was a beginner with no deployment experience and I am very lucky that I did not push a wrong button. All these stories of bill forgiveness reminds me of survivorship bias. Does this happens to everyone that reaches out to support or just the ones that get enough traction on social media? I am pretty sure there is no official policy from AWS, GCP or Azure.
themafia 1y ago
An alternative title might be "Failure to read the documentation horrors." If you didn't sit down with the documentation, the pricing guide, and a calculator before you decided to build something then you share a significant portion of the fault.
raw_anon_1111 1y ago
For everyone complaining about no free tier that blocks you from being charged https://aws.amazon.com/free/ https://aws.amazon.com/free/ Experience AWS for up to 6 months without cost or commitment Receive up to $200 USD in credits Includes free usage of select services No charges incurred unless you switch to the Paid Plan Workloads scale beyond credit thresholds Access to all AWS services and features
dang 1y ago
Related. Others? Single day Firebase bill for $100k - https://news.ycombinator.com/item?id=43884892 https://news.ycombinator.com/item?id=43884892 - May 2025 (14 comments) Serverless Horrors - https://news.ycombinator.com/item?id=39532754 https://news.ycombinator.com/item?id=39532754 - Feb 2024 (169 comments)
jiggawatts 1y ago
I keep telling customers: "The cloud will scale to the size of your wallet." They don't understand what I mean by that. That's okay, they'll learn! Anyway, this kind of thing comes up regularly on Hacker News, so let's just short-circuit some of the conversations: "You can set a budget!" -- that's just a warning. "You should watch the billing data more closely!" -- it is delayed up to 48 hours or even longer on most cloud services. It is especially slow on the ones that tend to be hit the hardest during a DDoS, like CDN services. "You can set up a lambda/function/trigger to stop your services" -- sure, for each individual service, separately, because the "stop" APIs are different, if they exist at all. Did I mention the 48 hour delay? "You can get a refund!" -- sometimes, with no hard and fast rules about when this applies except for out of the goodness of some anonymous support person's heart. "Lots of business services can have unlimited bills" -- not like this where buying what you thought was "an icecream cone" can turn into a firehouse of gelato costing $1,000 per minute because your kid cried and said he wanted more. "It would be impossible for <cloud company> to put guardrails like that on their services!" -- they do exactly that, but only when it's their money at risk. When they could have unlimited expenses with no upside, then suddenly, magically, they find a way. E.g.: See the Azure Visual Studio Subscriber accounts, which have actual hard limits. "Why would you want your cloud provider to stop your business? What if you suddenly go viral! That's the last thing you'd want!" -- who said anything about a business? What if it's just training? What if your website is just marketing with a no "profit per view" in any direct sense?
tempodox 1y ago
How are all these cases of exorbitant surprise bills not prosecuted as fraud?
chmod775 1y ago
These guys charge $550 for a measly terabyte of bandwidth? If you get a dedi on a 10Gb/s guaranteed port and it works out to more than $3 / TB, you're probably getting scammed. How does "serverless" justify 150x that? Are people hosting some silly projects really dense enough to fall for that kind of pricing? Just get a $10 VPS somewhere or throw stuff on GH pages. Your video game wiki/technical documentation/blog will be fine on there and - with some competent setup - still be ready for 10k concurrent users you'll never have.
deleted 1y ago
[deleted]
cgijoe 1y ago
Title should really be "Cloud Hosting Horrors", not serverless per se.
ghost_co 1y ago
Does anyone heard a success stories from cloud usage? Really, we(they) in the company decided to move in cloud everything from on-prem, it should save costs say them. But, as result you anyway need DevOps, some complications with development, local environments and not only. For not short career I faced some good examples, but it's more about unique situations, not a rule and a lot companies continue pay a lot for some small bunch of utility. Maybe I'm wrong, but such topics about this hell heard a lot of timesand only on some conference: success stories (because they should say: success)
rjakobsson 1y ago
Hahaha, this is awesome!
gangtao 1y ago
I got a large GCP logstorage bill as my applicatino is writing too much logs

626 points by operator-name 1y ago

After a quick check on Vercel stories, it seems all payments were discarded or mistakes in the first place. Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?

Alifatisk 1y ago
> Do you need to tweet about it to be reimbursed? This is what scares me, is social media the only way to get things sorted out nowadays? What if I don't have a large following nor an account in the first place, do I have to stomach the bill?
- wg002 1y ago
  I can't imagine them sending it to collections. What kind of recourse would a company like Vercel have if you don't pay it?
  - immibis 1y ago
    They can sue you for the bill plus the legal costs.
- pelagicAustral 1y ago
  This is exactly what happened to me during Covid... I had a flight that got cancelled at the beginning of the pandemic since the country closed the orders (essentially). A year after, still on lock downs and et al, I wanted to enquire about a refund, for months I got not answer, until I caught wind that people using Twitter were actually getting results. Now, I don’t use social media at all, so I had to create a Twitter account, twit about my case et voila! 30 mins after I got a response and they send me a PM with a case number... Not even going to mention the airline, but it is infuriating...
pjmlp 1y ago
No, at least in enterprise consulting for these kind of hosting, usually there is a contact person on the support team that one can reach directly. However these projects are measured in ways that make Oracle licenses rounding errors. Which naturally creates market segmentation on who gets tier 1 treatment and everyone else.
- viraptor 1y ago
  Once you're in a contract + TAM territory, pricing works very differently. Also, temporary experiments and usage overruns become an interesting experience where the company may just forget to bill you a few thousands $ just because nobody looked at the setup recently. Very different situation to a retail user getting unexpected extra usage.
deleted 1y ago
[deleted]
Havoc 1y ago
Relying on the mercy of a support agent that may be having a bad day is a poor strategy
tonyhart7 1y ago
I mean if developer got charged with 100k, more often than not the bank would decline that first maybe if you didn't have that high credit limit but what happen if this happen to corporate account and somewhere resource get leaked??? multi billions dollar company probably just shrug it off as opex and call it a day
interloxia 1y ago
Someone at a community group I'm in messed up playing with Azure through their free for non-profits offering^. We were out about 1.2k€. Not huge but huge for us. Encouraged by comments on HN over the years I had them ask support to kindly to wave it. After repeating the request a few times they eventually reduced their bill to <100€ but refused to wave it entirely. So even without shaming on social media, But it probably depends. It's worth at least asking. ^The deal changed about six months ago.
- cuu508 1y ago
  It's waive, not wave

joshstrange 1y ago

I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one: https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-can-make-your-aws-bill-explode-934a383cb8b1 https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c... About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.

siva7 1y ago
How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
sherburt3 1y ago
I believe they changed that shortly after that blog post went viral: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-no-charges-several-http-error-codes/ https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
- franktankbank 1y ago
  Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
  - pooper 1y ago
    > Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great! It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior". > AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception. The stench of this bovine excrement is so strong that it transcends space time somehow.

wg002 1y ago

This site is a bit dated. I remember in response to this Vercel added a way to pause your projects when hitting a spend limit. I enabled it for my account. Still, it made me question why I'm not using a VPS.

aurareturn 1y ago
Vercel used to be called Zeit. They had a server product called Now that gave you 10 1CPU/1GPU instances for $10/month (or $20 I forgot). It was the best deal. When Vercel switched everything to serverless, it all became pretty terrible. You need 3rd party services for simple things like DB connection pooling, websockets, cron jobs, simple queue, etc because those things aren’t compatible with serverless. Not to mention cold starts. Just a few weeks ago, I tried to build an API on Next.js+Vercel and get random timeouts due to cold start issues. Vercel made it easier to build and deploy static websites. But really, why are you using Next.js for static websites? Wordpress works fine. Anything works fine. Serverless makes it drastically harder to build a full app with a back end.
- pjmlp 1y ago
  Serverless is the most common deployment on MACH projects. Because when everything is a bunch of SaaS Lego bricks, serverless is all one needs for integration logic, and some backend like logic. Add to it that many SaaS vendors in CMS and ecommerce space have special partner deals with Vercel and Nelify. https://macharchitecture.com/ https://macharchitecture.com/

dakiol 1y ago

> I had cloudflare in front of my stuff. Hacker found an uncached object and hit it 100M+ times. I stopped that and then they found my origin bucket and hit that directly. Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?

solatic 1y ago
No. Your buckets should be private, with a security rule that they can only be accessed by your CDN provider, precisely to force the CDN to be used.
- rwmj 1y ago
  Why isn't that the default? I'm glad I use a Hetzner VPS. I pay about EUR 5 monthly, and never have to worry about unexpected bills.
  - graemep 1y ago
    Because not all uses for buckets fit that. Buckets are used for backups, user uploads, and lots of things other than distributing files publicly.
    - graemep 1y ago
      I would say its probably not a good idea to make a bucket directly publicly accessible, but people do not do that. A lot of the point of serverless is convenience and less admin and things like adding a layer in front of the bucket that could authenticate, rate limit etc. is not convenient and requires more admin.
  - kdps 1y ago
    Don't they charge for every TB exceeding the included limit? (website says "For each additional TB, we charge € 1.19 in the EU and US, and € 8.81 in Singapore.")

the__alchemist 1y ago

"Serverless" is a an Orwellian name for a server-based system!

magnusm 1y ago
Thats true!
Biganon 1y ago
"There's no cloud; it's just someone else's computer"
Spivak 1y ago
But your so called "no-code" system runs on code. Checkmate atheists. There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
- StevenWaterman 1y ago
  and your wireless modem has wires
- zahlman 1y ago
  In the "no-code" system, the end user does not write code. In the "serverless" system, the end user does connect to a server. It doesn't just "have" servers; they aren't a hidden implementation detail. Connecting to a website is an instrumental part of using the software.
  - viccis 1y ago
    "Serverless" refers to the demarcation point in the shared responsibility model. It means there aren't any servers about as much as "cloud hosting" means the data centers are flying.
  - immibis 1y ago
    In the "code" system the end user does not write code either - that's the developer's job. In the "no code" system it's the developer who doesn't write code, and in the "serverless" system it's the developer who doesn't set up servers.

zkmon 1y ago

Maintaining your own containers or VMs is hard considering how much risk appetite you have for the issues at infra level. So, yeah, when you complain about the costs of serverless, you are just paying for your low risk appetite low cost of your IT management.

paseante 1y ago

Yeah I also left my website hosted on Google Cloud because costs popped from everywhere, and there is basically no built-in functionality to limit them. So I didn't really slept relaxed (I actually slept great, but I hope you get the point) knowing that a bug could cost me... who knows how much. Actually, as the website of OP says, for spending control you have budget notifications and with that you can disable the billing for all the project altogether through some API call or something, I don't remember exactly, that is all there is. But still it looks like this functionality is just not there.

mortsnort 1y ago
You can write Google cloud functions to disable your credit card when certain thresholds are met pretty easily, but it's unethical that this isn't just a toggle somewhere in settings.
- Cpoll 1y ago
  Does that actually stop the spend immediately? If not, you're still on the hook for the bill. I suppose you can walk away and let them try to come after you, but that wouldn't work for a company.

EGreg 1y ago

Are there any protections these days at the cloud provider level? Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?

bc569a80a344f9c 1y ago
Not _really_. AWS has a budget tool, but it doesn’t natively support shutting down services. Of course, you can ingest the alerts it sends any way you want, including feeding them into pipelines that disable services. There’s plenty of blueprints you can copy for this. More seriously - and this is a legitimate technical limitation - of course AWS doesn’t check each S3 request or Lambda invocation against your budget, instead, it consolidates periodically via background reporting processes. That means there’s some lag, and you are responsible for any costs incurred that go over budget between such reporting runs.
- chuckadams 1y ago
  > of course AWS doesn’t check each S3 request or Lambda invocation against your budget If it can bill them per-invocation, why can't it also check against a budget? I don't expect it to be synchronous, but a lag of minutes to respond is still better than nothing. Can you even opt-in to shutting down services from the budget tool, or is that still something you have to script by hand from Cloudwatch alarms?
  - bc569a80a344f9c 1y ago
    You script it by hand. I think figuring out how to do this faster is less trivial than it might sound. I agree that synchronous checks aren’t reasonable. But let’s take Lambdas. They can run for 15 minutes, and if you consolidate within five minutes after a resource has been billed, that gives you a twenty minute lag. I’m not trying to make apologies for Amazon, mind you. Just saying that this isn’t exactly easy at scale, either. Sure, they bill by invocation, but that’s far from synchronous, too. In fact, getting alerts might very well be happening at the frequency of billing reconciliation, which might be an entirely reasonable thing to do. You could then argue that that process should happen more frequently, at Amazon’s cost.

Havoc 1y ago

Putting any sort of pay per use product onto the open internet has always struck me as insane. Especially with scaling enabled. At least stick a rate limited product in front of it to control the bleed. (And check whether the rate limit product is in itself pay per use...GCP looking at you)

deleted 1y ago

[deleted]

omnicognate 1y ago

It would help to round to the cent. With 3 digits to the right of the dot it's ambiguous whether it's a decimal point or a thousands separator, and the font and underline makes the comma vs dot distinction a bit unclear.

thedanbob 1y ago
A number of the titles appear to have 69 or 420 cents added to the amount that appears in the story.

jamil7 1y ago

Don’t most of these services have config options to protect against doing this? I haven’t used most of these services but it running up a bill during traffic spikes but not going down seems like it’s working as intended?

swiftcoder 1y ago
Nope, basically none of these services have a way to set a hard budget. They let you configure budget warnings, but it’s generally up to you to login and actually shut down everything to prevent from being billed for overages (or you have to build your own automation - but the billing alerts may not be reliable)
hxtk 1y ago
I know AWS in particular does not because they do not increment the bill for every request. I don't know exactly how they calculate billing, but based on what I do know about it, I imagine it as a MapReduce job that runs on Lambda logs every so often to calculate what to bill each user for the preceding time interval. That billing strategy makes it impossible to prevent cost overruns because by the time the system knows your account exceeded the budget you set, the system has already given out $20k worth of gigabyte-seconds of RAM to serve requests. I think most other serverless providers work the same way. In practice, you would prevent such high traffic spikes with rate limiting in your AWS API Gateway or equivalent to limit the amount of cost you could accumulate in the time it takes you to receive a notification and decide on a course of action.

ChrisMarshallNY 1y ago

At one time, I considered using Firebase as a backend, but then, I kept reading stories like these, and decided to roll my own. I'm fortunate to be able to do that. It's kind of amazing, though. I keep getting pressure from the non-techs in my organization to "Migrate to the Cloud." When I ask "Why?" -crickets. Industry jargon has a lot of power. Seems to suck the juice right out of people's brains (and the money right out of their wallets).

phoenixhaber 1y ago

When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money. I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.

croes 1y ago
That’s why I prefer prepaid cards or those I can easily freeze to prevent any booking.
- jsheard 1y ago
  If your card is declined and they don't feel like forgiving the bill, won't they just send debt collectors after you instead?
  - croes 1y ago
    Yes, but it's better they need to get their money than you need to get your money back. 100.000 easily can put you in ruining debt. It's the better position to still have your money even if you have to pay.
- kleinsch 1y ago
  Freezing a card doesn’t mean the debt is erased. They can still take you to collections.

nurettin 1y ago

I've had this twice. Once with oracle, once with azure. They both charged me $2000-$5000 for simply opening and closing a database instance (used only for a single day to test a friend's open source project) To be fair, support was excellent both times and they waived the bills after I explained the situation.

qcnguy 1y ago
How did you run up a $5000 bill for just testing a project? What kind of project was it that could put so much load on the DB?
- deleted 1y ago
  [deleted]

caboteria 1y ago

The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right? I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.

hvb2 1y ago
You don't think this happens on prem? Servers running an application that is no longer used? Sure they're probably VMs but their cost isn't 0 either
- sgarland 1y ago
  With that model, your cost doesn't change, though. When/if you find you need more resources, you can (if you haven't been doing so) audit existing applications to clear out cruft before you purchase more hardware.
  - remus 1y ago
    That's the equivalent of saying "just audit your cloud usage and remove stuff that's no longer used".
  - hvb2 1y ago
    The cost of going through that list often outweighs the cost of the hardware, by a lot. And in a lot of cases it's hard to find out if a production application can be switched off. Since the cost is typically small for an unused application, I don't know if there are many people willing to risk being wrong
    - sgarland 1y ago

fnord77 1y ago

looking forward to the "LLM token horrors" version

mdaniel 1y ago
I thought there was an OWASP for "denial of wallet" vulnerabilities but this link was the closest one I found https://www.prompt.security/vulnerabilities/denial-of-wallet-service https://www.prompt.security/vulnerabilities/denial-of-wallet... (although the link makes it sound like they're offering denials)
- kjs3 1y ago
  We call them "Billing Fatigue Attacks". Keep an eye on your autoscaling, kids...

api 1y ago

There should also be a general category for "cloud horrors" for things that cost $50k/month to host that would be $1500/month on a bare metal provider like Datapacket or Hetzner. I'm old enough to remember when cloud was pitched as a big cost saving move. I knew it was bullshit then. Told you so.

nchmy 1y ago
even $1500/mo on hetzner is a seriously large app. You could get 300 cpus and 1.5TB of RAM for that price.

emseetech 1y ago

This is why when I contract for an early stage startup, I pose the question: "What if your app went viral and you woke to a $20k cloud bill? $50k? $80k?" If the answer is anything less than "Hell yeah, we'll throw it on a credit card and hit up investors with a growth chart" then I suggest a basic vps setup with a fixed cost that simply stops responding instead. There is such a thing as getting killed by success and while it's possible to negotiate with AWS or Google to reduce a surprise bill, there's no guarantee and it's a lot to throw on a startup's already overwhelming plate. The cloud made scaling easier in ways, but a simple vps is so wildly overpowered compared to 15 years ago, a lot of startups can go far with a handful of digitalocean droplets.

mococa 1y ago

Couple years ago I was charged in USD 4K on Google Cloud after trying recursive cloud functions. I told them that was a mistake and they forgot the debit, they just asked to no do again.

game_the0ry 1y ago

I have a feeling I will be downvoted for this, but... Have the people posting these horror stories never heard of billing alerts?

skippyboxedhero 1y ago
Many of the stories on the site are from people who have billing alerts. If you have bot spam, how do you actually think their billing alerts work? The alert is updated every 100ms and shuts off your server immediately? That isn't how billing alerts can or should work.
- McGlockenshire 1y ago
  Yes, actually, if continuing to run the service is going to exceed my available budget then I do want the service turned off! If I can't pay for it, and I know I can't pay for it, what other possible choice do I have? Do any of you people have budgets, or do you all rely on the unending flow of VC money?
  - skippyboxedhero 1y ago
    That isn't how this can work. If you are running a service and then find out that AWS is spamming you every 100ms to find out what your CPU is doing (or calling out every 100ms) then people would be quite unhappy. The majority of these massive bills are due to traffic, there is pretty much no way that AWS could stop your server in time...if they had the choice, which they don't. I think my original point was unclear: I am pointing out that if you just think about how this stuff can possibly work, billing alerts can not work in the way you expect. The alert is updated async, the horse has bolted and you are trying to shut the gate. I don't use AWS for personal stuff because I know their billing alerts won't stop me spending a lot. Don't use them if that is a concern. I do use AWS at work, we are a relatively big customer and it is still very expensive for what it is. The actual hardware is wildly overpriced, their services aren't particularly scalable (for us), and you are basically paying all that overage for network...which isn't completely faultless either. Imo, using them in a personal capacity is a poor idea.

johnebgd 1y ago

We are building bare metal for our workloads… I don’t care if cloud is supposed to be cheaper because it never is. You can get a decent small business firewall to handle 10gbit fiber for $600 from unifi these days. Just another reason I’m glad I moved out of the Bay Area and nyc to a midwestern town for my company. I have a basement and can do rad things in my house to grow my business.

nchmy 1y ago
bUt wuT aBowT deV OpS?!

1oooqooq 1y ago

last employer asked for an estimate to migrate to cloud. it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr. ceo told to go ahead anyway (turn out company was being sold to Apollo) first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one. can't imagine how companies miss that step.

skippyboxedhero 1y ago

Hetzner, 16TBx2 HDD, 1TBx2 SDD, 64GB RAM, 20TB free bandwidth, $70/month. I used 1TB of traffic on a micro instance and it cost me $150 (iirc). Doesn't have to be this way.

shayway 1y ago

I guess I'm missing something, why is this 'serverless' horrors? If anything it seems to specifically be serverful horrors.

o11c 1y ago
"Serverless" is just marketing-speak for "somebody else's server".

stressback 1y ago

I read a lot of the posts at the little blog here and, uh, every single one sounds like a complete amateur making a cloud configuration mistake. I haven't found one that is the provider's fault or the fault of "serverless" I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself. Serverless isn't a horror, serverlesshorrors poster. You are the horror. You suck at architecting efficient & secure systems using this technology, you suck at handling cloud spend, and you suck at taking responsibility when your "bug" causes a 10,000x discrepancy between your expected cost and your actual bill. Just because you don't understand it doesn't mean it sucks

Cpoll 1y ago
You're not wrong about cloud configuration mistakes, but a tool that lets you increase costs 10000x (without even letting you set a safety) is a hell of a chainsaw. I'm more worried about the overconfident SRE that doesn't stay up at night worrying about these.
Bluecobra 1y ago
That being said, the cloud providers could do a better job explaining to new/naive users that great power comes with great responsibility and there is no hand holding. Someone might be more hesitant to willy nilly spin up something if a wizard estimates that the maximum cost could be $X per month.
webdevver 1y ago
truth nuke
McGlockenshire 1y ago
> every single one sounds like a complete amateur making a cloud configuration mistake Golly if only the configuration wasn't made this way on purpose exactly to cause this exact problem.
bhk 1y ago
Consider this analogy: Instead of using a root command shell, it is wise to use an account with appropriately restricted capabilities, to limit the downsides of mistakes. Cloud services support the notion of access control, but not the notion of network resource usage limits. It's an architectural flaw. Or do you always log in as root, like a real man, relying purely on your experience and competence to avoid fat-finger mistakes?

luxuryballs 1y ago

there should be some kind of insurance for bugs that introduce unusually expensive usage

mdaniel 1y ago
I believe any such policy would need its premiums based on the services used (and likely the qualifications of the staff) since, unlike rebuilding a house, the financial risk is almost unlimited with out of control cloud spend It reminds me of the Citi(?) employee who typed the wrong decimal place in a trade: computers make everything so easy!

deleted 1y ago

[deleted]

NullCascade 1y ago

Troy Hunt and HIBP is a good example in the other direction but Hunt has also been burned plenty of times by serverless. https://www.troyhunt.com/closer-to-the-edge-hyperscaling-have-i-been-pwned-with-cloudflare-workers-and-caching/ https://www.troyhunt.com/closer-to-the-edge-hyperscaling-hav...

dinvlad 1y ago

Andras (author of Serverless Horrors) knows what he’s talking about. The amount of brainwashing that big cloud providers have done, is insane.

petralithic 1y ago

This is some good marketing for Coolify, which the author makes as an open source platform as a service. I prefer Dokploy these days though, since it seems to be less buggy, as Coolify seems to have such bugs due to being on PHP. https://coolify.io/ https://coolify.io/ https://dokploy.com/ https://dokploy.com/

khromov 1y ago
CapRover is another good alternative, and also much more lightweight than Coolify, easily runs on even a 512MB server: https://caprover.com/ https://caprover.com/

andrasbacsai 1y ago

look mom & dad, I am famous!

adamddev1 1y ago

I remember at the beginning of the serverless hype how they said it was great because it automatically scaled as big as you need it. Given how sudden and massive these "scaling spikes" can be, I would much rather deal with a death-hugged VPS than a $100k bill. Plus the VPS is just so much faster in most cases.

acoustics 1y ago

The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent. Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build. I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend? Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?

rr808 1y ago
Maybe, but its a huge reason to use real servers instead of serverless.
- pixl97 1y ago
  I mean real servers get hit with things like bandwidth fees so it's not a 100% solution.
  - estimator7292 1y ago
    Not even remotely the same scale of problem. Like at all. If your business suddenly starts generating Tbs of traffic (that is not a ddos), you'd be thrilled to pay overage fees because your business just took off. You don't usually get $10k bandwidth fees because your misconfigured service consumes too much CPU. And besides that, for most of these cases, a small business can host on-prem with zero bandwidth fees of any type, ever. If you can get by with a gigabit uplink, you have nothing to worry about. And if you're at the scale where AWS overages are a real problem, you almost certainly don't need more than you can get with a surplus server and a regular business grade fiber link. This is very much not an all-or-nothing situation. There is a vast segment of industry that absolutely does not need anything more than a server in a closet wired to the internet connection your office already has. My last job paid $100/mo for an AWS instance to host a GitLab server for a team of 20. We could have gotten by with a junk laptop shoved in a corner and got the exact same performance and experience. It once borked itself after an update and railed the CPU for a week, which cost us a bunch of money. Would never have been an issue on-prem. Even if we got DDoSed or somehow stuck saturating the uplink, our added cost would be zero. Hell, the building was even solar powered, so we wouldn't have even paid for the extra 40W of power or the air conditioning.

princevegeta89 1y ago

In my experience: Fuck serverless. If we're building anything bigger than a random script that does a small unit of work, never go for serverless. A company I recently worked for went with Serverless claiming that it would be less maintenance and overhead. It absolutely was the worst thing I've ever seen at work. Our application state belonged at different places, we had to deal with many workarounds for simple things like error monitoring, logging, caching etc. Since there was no specific instance running our production code there was no visibility into our actual app configuration in production as well. Small and trivial things that you do in a minute in a platform like Ruby on Rails or Django would take hours if not days to achieve within this so-called blistering serverless setup. On top of it, we had to go with DB providers like NeonDb and suffer from a massive latency. Add cold starts on top of this and the entire thing was a massive shitshow. Our idiot of a PM kept insisting that we keep serverless despite having all these problems. It was so painful and stupid overall.

kikki 1y ago
Why was your PM making tech decisions?
- Nextgrid 1y ago
  Looks like you need the "quiet part" said out loud: Chances are, the company was fishing for (or at least wouldn't mind) VC investment, which requires things being built a certain (complex and expensive) way like the top "startups" that recently got lots of VC funding. Chances are, the company wanted an invite to a cloud provider's conference so they could brag about their (self-inflicted) problems and attract visibility (potentially translates to investment - see previous point). Chances are, a lot of their engineering staff wanted certain resume points to potentially be able to work at such startups in the future. Chances are, the company wanted some stories about how they're modern and "cloud-native" and how they're solving complex (self-inflicted) problems so they can post it on their engineering blog to attract talent (see previous point). And so on.
  - princevegeta89 1y ago

zahlman 1y ago

I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work. To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an Internet connection, it's not to send data to a specific place, under the developer's control, for the purpose of making the software system function as advertised.

fritzo 1y ago
It's like a company with no employees. There are still people performing services, but on temporary contracts.
maxbond 1y ago
That's generally called "local." Serverless is poorly named but describes how certain backends are deployed, not applications without a backend.
leptons 1y ago
A "Server" is typically a single machine that has a specific OS and runs layers of various software that allows your business logic to be accessed by other computers (by your users). For a "Server" you typically have to choose an OS to run, install all the support software (server monitoring, etc), update the software, and if the server fails you have to fix it or rebuild it. With "Serverless", your code is in a "function as a service" model where all you have to worry about is the business logic (your code). You don't have to set up the server, you don't have to install the server OS, or any basic server software that is needed to support the business logic code (http server, etc). You don't have to update the server or the underlying server software. You don't have to perform any maintenance to keep the server running smoothly. You never (typically) have to worry about your server going down. All you have to do is upload your business logic function "somewhere" and then your code runs when called. Essentially you do not have to deal with any of the hassle that comes with setting up and maintaining your own "server", all you have to do is write the code that is your business logic. That's why it's called "Serverless" because you don't have to deal with any of the hassle that comes with running an actual "server".

jppope 1y ago

This is a weird take on an incredibly useful paradigm (serverless). One the one side, there are obviously precautions that all of these users could have taken to avoid these charges on the other hand its totally common to spin up a thing and forget about it or not do your due diligence. I totally feel for the people who have been hit with these chargers. At the end of the day though the whole think feels like a carpenter shooting themselves in the foot with a nail gun then insisting that hammers are the only way to do things.

mahirsaid 1y ago

Seem likes there are mistakes that were made on behalf of the users. The attackers found these mistakes and took advantage of them. i don't think "severless" is the problem.

bapak 1y ago
Serverless is the problem in that most serverless services don't let you hard-cap spend. This issue is serverless-specific. If I pay $20/month on VPN the most frightening thing that can happen is the client calling you about your website being down, not a $100k bill.

general1465 1y ago

I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything. That's just a problem waiting to happen while you are always running tests on production...

scarface_74 1y ago
This is nowhere near being true.
nostrebored 1y ago
1. Put your stuff in a stack. Deploy it to your isolated developer account. Basically free staging environment. 2. Use the officially supported docker runtime for local testing. 3. Treat it like any other code and make unit tests 4. Use one of the tools like localstack to emulate your staging env on your machine. There are so many options that I don’t know how you could walk away with your opinion.
- Nextgrid 1y ago
  Or you could just write conventional software. But I get it, you don't get resume points nor invites to cloud-provider conferences for that. > Basically free staging environment. [emphasis mine] Not really. Sure, the cost would usually be peanuts... until you have an infinite loop that recursively calls more lambdas. Then you have a huge bill (but hey that pays for your invites to their conferences, so maybe it's a blessing in disguise?). And yes, you will pretty much always get it refunded, but it's still a hassle and something that is absolutely not necessary. Snark aside, having an opaque dev environment always constrained by bandwidth and latency that can’t be trivially backed up/duplicated is a terrible idea and why I always recommend against “serverless”, even besides the cost concerns. Serverless is OK for small, fully self contained pieces or code that are fire and forget. But for anything complex that’s likely to require maintenance, no thanks.
- rurp 1y ago
  Eh, I worked on a large serverless project that worked hard to follow best practices but it was still very clunky to run and test code locally. The local serverless tools simply didn't work for our project and they had so many limitations I'm skeptical they work for most non-prottypes. Deploying a stack to your own developer environment works fine and is well worth doing, but the turnaround time is still painful compared to running a normal web framework project locally. Deploying a stack takes much much longer than restarting a local server. Serverless isn't all bad, it has some nice advantages for scaling a project, but running and debugging a project locally is a definite weak spot.

CivBase 1y ago

I can't imagine hosting a small-time project on rented infrastructure without some kind of mechanism to terminate it once costs exceed a reasonable threshold.

mitjam 1y ago

I once found an official Microsoft example repo to deploy an LLM gateway on Azure with ALB. Glad I did the tedious work of estimating the costs before I hit the deploy button (had to go though many Biceps manifests for that). The setup would have cost me about 10k/month.

nromiun 1y ago

I was also too careless with AWS when I was a beginner with no deployment experience and I am very lucky that I did not push a wrong button. All these stories of bill forgiveness reminds me of survivorship bias. Does this happens to everyone that reaches out to support or just the ones that get enough traction on social media? I am pretty sure there is no official policy from AWS, GCP or Azure.

themafia 1y ago

An alternative title might be "Failure to read the documentation horrors." If you didn't sit down with the documentation, the pricing guide, and a calculator before you decided to build something then you share a significant portion of the fault.

raw_anon_1111 1y ago

For everyone complaining about no free tier that blocks you from being charged https://aws.amazon.com/free/ https://aws.amazon.com/free/ Experience AWS for up to 6 months without cost or commitment Receive up to $200 USD in credits Includes free usage of select services No charges incurred unless you switch to the Paid Plan Workloads scale beyond credit thresholds Access to all AWS services and features

dang 1y ago

Related. Others? Single day Firebase bill for $100k - https://news.ycombinator.com/item?id=43884892 https://news.ycombinator.com/item?id=43884892 - May 2025 (14 comments) Serverless Horrors - https://news.ycombinator.com/item?id=39532754 https://news.ycombinator.com/item?id=39532754 - Feb 2024 (169 comments)

jiggawatts 1y ago

I keep telling customers: "The cloud will scale to the size of your wallet." They don't understand what I mean by that. That's okay, they'll learn! Anyway, this kind of thing comes up regularly on Hacker News, so let's just short-circuit some of the conversations: "You can set a budget!" -- that's just a warning. "You should watch the billing data more closely!" -- it is delayed up to 48 hours or even longer on most cloud services. It is especially slow on the ones that tend to be hit the hardest during a DDoS, like CDN services. "You can set up a lambda/function/trigger to stop your services" -- sure, for each individual service, separately, because the "stop" APIs are different, if they exist at all. Did I mention the 48 hour delay? "You can get a refund!" -- sometimes, with no hard and fast rules about when this applies except for out of the goodness of some anonymous support person's heart. "Lots of business services can have unlimited bills" -- not like this where buying what you thought was "an icecream cone" can turn into a firehouse of gelato costing $1,000 per minute because your kid cried and said he wanted more. "It would be impossible for <cloud company> to put guardrails like that on their services!" -- they do exactly that, but only when it's their money at risk. When they could have unlimited expenses with no upside, then suddenly, magically, they find a way. E.g.: See the Azure Visual Studio Subscriber accounts, which have actual hard limits. "Why would you want your cloud provider to stop your business? What if you suddenly go viral! That's the last thing you'd want!" -- who said anything about a business? What if it's just training? What if your website is just marketing with a no "profit per view" in any direct sense?

tempodox 1y ago

How are all these cases of exorbitant surprise bills not prosecuted as fraud?

chmod775 1y ago

These guys charge $550 for a measly terabyte of bandwidth? If you get a dedi on a 10Gb/s guaranteed port and it works out to more than $3 / TB, you're probably getting scammed. How does "serverless" justify 150x that? Are people hosting some silly projects really dense enough to fall for that kind of pricing? Just get a $10 VPS somewhere or throw stuff on GH pages. Your video game wiki/technical documentation/blog will be fine on there and - with some competent setup - still be ready for 10k concurrent users you'll never have.

deleted 1y ago

[deleted]

cgijoe 1y ago

Title should really be "Cloud Hosting Horrors", not serverless per se.

ghost_co 1y ago

Does anyone heard a success stories from cloud usage? Really, we(they) in the company decided to move in cloud everything from on-prem, it should save costs say them. But, as result you anyway need DevOps, some complications with development, local environments and not only. For not short career I faced some good examples, but it's more about unique situations, not a rule and a lot companies continue pay a lot for some small bunch of utility. Maybe I'm wrong, but such topics about this hell heard a lot of timesand only on some conference: success stories (because they should say: success)

rjakobsson 1y ago

Hahaha, this is awesome!

gangtao 1y ago

I got a large GCP logstorage bill as my applicatino is writing too much logs