11 ms·

Serverless Horrors

292 points by franciscop 3y ago

BizarroLand 3y ago
Minor criticism, I get that this is a page that collects links to other pages talking about the horrors of serverless. Sure. Makes perfect sense. I would like to see some small amount of commentary on each page that covers what I am about to read. Not much, but half a paragraph of introduction would be nice.
remram 3y ago
Is there the opposite website, listing cloud services that do have spend limits?
rldjbpin 3y ago
in terms of web hosting, why does hosting using serverless / paas means not using a cdn? it also baffles me that these services cannot tell between legitimate traffic and ddos.
crooked-v 3y ago
Also see the Netlify comment thread linked from one of the Reddit posts (https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-billing-caused-by-ddos/13086 https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...), where one of their representatives specifically says that if you suffer a DDOS they will do nothing to stop you from getting a bajillion-dollar bandwidth bill, even if you're on the "free" tier.
- voidwtf 3y ago
  That's my favorite thing about all these services. None of them have an equivalent of a stop-loss. They'll notify you IF you setup an alert, but that's it. Often times the alerts are way after the damage is done, with some billable metrics often lagging in reporting/alerting by hours. The default should be safe with easy options to increase the limits.
  - jmathai 3y ago
    Lambda and GCP Cloud Functions do have cost controls. You can specify the maximum number of instances that can be spun up at any given point. I think the UX can make it more friendly to specify the types of traffic and cost you’re expecting but most erroneous spikes are the result of a customer misconfiguration or a legitimate spike in traffic. It’s the former which are problematic for customers. They can request a credit and we (GCP) try and accommodate. It’s not a feature designed for profit. But it also tends to fall below the cut line for prioritization to fix. But it’s in the backlog and doesn’t go unnoticed. (Former PM on GCP Cloud Functions)
    - sroussey 3y ago
      Do users setup cost controls before activating the service? Or is this something they need to know how to do and do on their own?
      - jmathai 3y ago
        Separate from billing, the product-native cost controls which prevent over-consumption is configured at the resource level. gcloud run deploy ..... --max-instances=N
  - fragmede 3y ago
    What you really want is prepaid instead of post-paid billing. Charge up the account with $100, and stop working and delete my data if that gets hit, with an auto renew option to recharge some amount every month.
    - voidwtf 3y ago
      That seems a little reductive. I don’t think it’s unreasonable to say, my current spend is ~13k a month. I should be able to set a 20k cap, such that non fixed billable like bandwidth or map tiles get paused while someone determines whether we’d like to proceed at that rate. There is a not so fine line between scaling for a rush/event and a resume generating event.
- huijzer 3y ago
  I wonder whether they are running out of money which then causes these things. Netlify used to be a very nice and unique service, but then came GitHub, GitLab and Cloudflare Pages who all provide roughly the same services for less money.
- hamoodhabibi 3y ago
  It used to be we will take your server down. Now its we'll DDOS your serverless website and leave you a 100k bill. I'm not sure how sustainable such business model is. When you owned the server, you could unplug it. Now you have no way of knowing if somebody is going to hit your /api a million times per minute
  - andrasbacsai 3y ago
    I also prefer to get a (decaf) coffee, listen to some music while someone DDoS'd my VPS. I prefer to pay few $ / month for my VPS instead of paying thousands and "survive" the DDoS.
    - s9df898r32h 3y ago
      I pay $10/year for my VPS and host a WordPress Woocommerce store on it... It doesn't get much traffic, but it didn't take long to pay for itself either
      - InvOfSmallC 3y ago
        where?
        s9df898r32h 3y ago
        [dead]
        Filligree 3y ago
        I explicitly block everything in Russia (and China and Hong Kong) from accessing my servers, because otherwise the rate of SSH attacks is so high it actually prevents SSH from working reliably -- the connection pool fills up. Just an FYI; I don't think it's that unusual. Blocking those geoips dramatically reduced my logging volume.
        3y ago
- ponector 3y ago
  Can you use free tier under a fake name? To drop ridiculous bills. How do they validate the customer?
  - throwaway11460 3y ago
    There is no free plan, there is just free limit - you must add a card.
    - s9df898r32h 3y ago
      just use a virtual card and then change the spending limit to $0? or freeze it
      - throwaway11460 3y ago
        That still passes your personal info.
        s9df898r32h 3y ago
        at least you wont get charged without your knowledge... which in the end, is the first step in getting this non-sense fixed, for a person... of course the government should fix it for everyone. This shit should probably be considered fraud anyways... companies get free passes all the time. My name (s9df898r32h) is my company, I should be able to steal whatever I want /s Why are such large companies so incompetent though...
        remram 3y ago
        You will have to pay it AND a fee for fraud, and possibly their legal fees for tracking you down. You also will have a difficult time arguing that you were in good faith with this kind of shenanigans. Bouncing checks are not a life hack to get stuff for free, they're a crime.
        3y ago
- grubbs 3y ago
  So what do I do if this happens to me on the free tier? Just never pay the bill and leave? Will they come after me?
LAC-Tech 3y ago
On the one hand I am glad to see sceptisim of serverless gaining mainstream acceptance. On the otherhand, I fear it will just turn into an unreasoned backlash against technology that is often useful if not as often as its early proponents said it would be (See NoSQL).
tisc 3y ago
Costing money is one thing, but not throttling a queue consuming serverless function and effectively DDOSsing yourself is the true horror story working with serverless :)
- bigiain 3y ago
  That's kinda not "server less" specific though. There's no end of stories of people discovering 10s to thousand in bill run up on AWS (ir Azure or GCP) because they foot gunned themselves and didn't set up alerts and/or automated service suspension using the cloud providers available tools. Part of me thinks you shouldn't be allowed to provision EC2/S3/Lambda/whatever without first having set up a Cloudwatch alert and a rule to stop your service if your bill goes past expected/acceptable limits. Of course Amazon (or Microsoft or Google) could make that _way_ easier than they do. It'd be nice if there was a simple form in your account setting where you could configure it to suspend services if your projected monthly bill passed $limit. I think it's a more generic "horror story of working with other people's computers".
usr1106 3y ago
How is no spending limit a serverless problem? It's a cloud problem. Edit: Well, if you have small cloud servers they might go down under an attack. And if you pay only outgoing like in AWS that's good for you.
- simonjgreen 3y ago
  It’s an architecture failing. Setting maximums is critical. If you choose a platform that doesn’t support that more fool you. It’s implicitly easier to have a ‘circuit breaker’ in a traditional environment, but still warrants consideration. Load test for success and failure.
- happymellon 3y ago
  It's not a cloud problem. The cloud is just somebody else's computer. It's either a billing or an architecture problem. Either let me throttle requests to prevent overbilling, or cut me off when I hit a certain value.
- Hamuko 3y ago
  Because serverless 1) uses pay-as-you-go more often than traditional models 2) scales better to achieve massive usage. If you DDoS my $5/month DigitalOcean VPS, it's not going to increase how much I have to pay DigitalOcean in terms of compute, and most likely the VPS will buckle under load before it can actually rack up massive transfer costs (since those are also pay-as-you-go for DO).
- lelanthran 3y ago
  > How is no spending limit a serverless problem? > It's a cloud problem. It's a serverless problem because it can be mitigated if using the cloud, but not mitigated if it is serverless. I have a cloud instance. I don't have this problem and never will. If I had serverless instead, I will have this problem.
000ooo000 3y ago
Something feels a bit off about the creator of this site also pushing a competitor. On one hand, they're upfront about it, and on the other.. they're upfront about it. At least it might prompt the companies in question to be a bit less scummy.
- andrasbacsai 3y ago
  Which competitor do I push exactly? :) (I created serverlesshorrors)
  - hurflmurfl 3y ago
    The link to the "competitor" is front and center on the site, so I imagine what you're really trying to say is that this is NOT a competitor for the following reasons: <missing>. Otherwise this seems like a bait question:)
    - d1sxeyes 3y ago
      Andras (who created both Serverless Horrors and Coolify, which I imagine you're describing as the competitor here) is a pretty solid guy from everything I've seen and I've never seen him ask bait questions or engage in bad faith in discussions here or on other platforms. Although he calls Coolify a Netlify 'alternative', it's 100% 'bring-your-own-server', and is not at all serverless. Coolify is basically a (very nice and featureful) fancy frontend to Docker with integration with Git. Coolify is free if you want to install it yourself and don't mind the (minimal) tinkering needed to get it set up and keep it up-to-date, although you can pay a monthly fee to have your own servers managed for you, the main benefits are either to support continued development, or to access priority support. You still need to bring your own server, even if you're paying the management fee. It's not a competitor to Netlify, it's an alternative.
      - hurflmurfl 3y ago
        Just to clarify, I didn't think that Coolify is a competitor. What I was getting at is that we now have a thread with 4 replies: (1) you pitch competing product (2) what product? (3) Coolify (4) it's not competitor because ... I just wanted to point out that comment #2 only seems to prompt question #3, whereas the helpful reply #4 created by yourself obviates the need for #2 and #3.
tutfbhuf 3y ago
I don't think the Netlify case has happend due to serverless architecture. Yes, there are many technical issues with serverless architectures, but getting a huge bill for incoming traffic is independent from a serverless architecture. You could also put your own hardware in a colo and pay for traffic. If your data center provider does not protect you from DDoS and simply bills by the TB, then the same situation would occur. Of course, one could argue that the prices per TB are much cheaper in a colo situation compared to Netlify and therefore a 100k bill is very unlikely to occur, but then I would make this the main point: absurdly high traffic costs and no DDoS mitigation, and wouldn't put it behind a "serverless horrors" title.
- adrianN 3y ago
  If your stuff scales to infinity your money burn rate also scales to infinity. A single server in a colo can only do so much damage to your wallet before it’s saturated.
  - tutfbhuf 3y ago
    Sure, but according to the article, he received the 100k bill because of peak traffic of 60TB of bandwidth per day over several days. If you have a dedicated server in a colo, then a 10GbE connection is very common these days. With a 10GbE connection, you can serve 100TB of traffic in a single day. Hence, if the colo had the same prices as Netlify for traffic, then a single server with a 10GbE connection would be sufficient for such a high bill, no serverless and infinity scaling required, just absurdly high traffic costs.
    - zrn900 3y ago
      Nah.You get a dedicated box at hetzner and get unmetered 1gbit and use as much as you want as long as you dont do mining or piracy etc.
    - albert180 3y ago
      No Colo charges the same insane prices for Traffic as Cloud Providers. Hetzner charges 1€/TB over 20TB for their 10GbE Servers
liampulles 3y ago
I don't think infinite scaling is desirable. You probably want to say "if the value or rate of increase in xyz" metric reaches a certain level, don't scale further, or start scaling down. As a requirement for a serverless deployment, not an option.
merelysounds 3y ago
A kill switch seems a practical and trivial solution. E.g. something toggled when a monthly bill exceeds a configurable amount. I’m guessing many hobby users would set that to $100, end of story. I don’t understand why it’s so unpopular to offer that. I’d imagine the providers would benefit too in the long run.
- andrasbacsai 3y ago
  It would be practical for the user, but not for the company. Why does this not exist after years (more than a decade since EC2) of cloud computing? Because it is not good for the VC investors.
  - catlifeonmars 3y ago
    AWS Lambda has had this knob for quite a few years, it’s called “provisioned capacity”, and it defines an upper bound to function concurrency.
    - merelysounds 3y ago
      While helpful, this is not the same. E.g. a global kill switch could be harder to overlook and easier to manage.
- webprofusion 3y ago
  You can set a hard budget on most cloud providers, including Azure etc. I actually did this thinking I was just setting a budget alert and that was a new and interesting way to interrupt production.
  - merelysounds 3y ago
    > You can set a hard budget on most cloud providers Note that the submission references serverless providers, specifically netlify and vercel. I don’t think these support an easy to configure kill switch - in any case it wasn’t easy to configure in the scenarios listed on the page.
jeroenvlek 3y ago
Seems very inspired by Molly White's page [0] [0] https://www.web3isgoinggreat.com/ https://www.web3isgoinggreat.com/
- KaiserPro 3y ago
  its in the style of long runnning dailyWTF: https://thedailywtf.com/ https://thedailywtf.com/ There used to be a whole host of specific fuckup like list site, sadly I can't find them anymore.
- rnts08 3y ago
  This is brilliant, there goes my afternoon. Thank you
shubhamjain 3y ago
Everyone in this thread is talking about how this is not a Serverless problem, but a cloud problem. That's correct, but the point stands. Bandwidth is pure margin and to me, it's a bit scummy to a) charge this much for bandwidth and b) offer no tools to combat attacks that aren't in the customer's control. According to this thread[1], Netlify doesn't even have an option to take a site down for a while if it's under DDoS attack. Netlify has multiple options to solve the problem: billing controls, request throttling, setting bandwidth limit, and waving of overages if it was a genuine DDoS, but it doesn't seem they are interested because that'd be like killing the golden goose. Just look at all the things bunny.net CDN offers [2]. It's disappointing how we acquiesce to explanations of cloud providers that are motivated by nothing other than greed. [1]: https://answers.netlify.com/t/limiting-bandwidth-traffic-to-netlify-on-starter-tier-plan/18163/8 https://answers.netlify.com/t/limiting-bandwidth-traffic-to-... [2]: https://support.bunny.net/hc/en-us/articles/360014190440-Understanding-Pull-Zone-Network-Limits https://support.bunny.net/hc/en-us/articles/360014190440-Und...
simonjgreen 3y ago
Bandwidth charging models from hosts are crazy. They don’t model their upstream costs in any way. Transit providers charge based on 95th% and ports scale based on highest burst not volume passed. I’ve always hated it.
SushiHippie 3y ago
Related submission from yesterday: Netlify just sent me a $104k bill for a simple static site | https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776
bArray 3y ago
> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud. But still this doesn’t invalidate the point of having protection against such attacks, and limit the spending. This is extremely sad. It's like we are taking steps backward. A 3.44MB should not be an issue and if it is, the answer should not be to host it elsewhere. If there is no other lesson learned here, it's that there is no such thing as 'free'. As others have said, there should be more done to prevent such large losses without some kind of limit. I would also like to point out that VPS's are extremely cheap [1] and extremely easy to manage. They have automatic limits. [1] https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/ https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/
- cpursley 3y ago
  VPSs being “easy to manage” is a strong option full of assumptions.
  - ies7 3y ago
    I remember a comment from this site. Something like: we must study how Amazon and other big cloud's marketing work so great to make this generation developers think that vps or selfhosting are hard. IMO vps was easy before and even easier now to manage.
    - kikimora 3y ago
      Please explain how you rotate ssh keys, store audit logs, backup (and test backup procedures), configure secure network between vpses such that your neighbor cannot eavesdrop.
      - deleted 3y ago
        [deleted]
      - bcaxis 3y ago
        * You don't have to rotate what doesn't get out. Limit ingress to relevant IPs reduces this surface area a lot. * SCP to a system built for storage. Not really essential for many systems - system logs are fine. * Every VPS provider comes with a backup check box. * Tailscale is really simple.
throwawayyy9237 3y ago
Perhaps I'm overreacting, but I had my personal site on Netlify and decided to move it. I don't need anything more than a dumping ground for HTML, and Netlify was "good enough" (I wasn't aware that this issue existed) What brought it closer to home was the characteristics of the recently affect site (same number of daily visits, not popular (a bit niche), etc) where similar to mine. Moving is easy enough, mostly requiring a DNS update, since I prefer to build the HTML locally and then just dumping it somewhere. What struck me as odd, when looking for an alternative is that almost none of the popular solutions Netlify, Vercel, Cloudfare (AFAIK) actually implement a spend limit. This seems like such a basic thing to do...
- aetherspawn 3y ago
  Cloudflare has DDOS protection which you can set to basically paranoid and once a DDOS starts then everyone will get a captcha. This limits your spend quite effectively.
  - Filligree 3y ago
    And also stop me accessing the site entirely. I don't know why, but every site with CF DDoS protection set to paranoid gives me infinite captchas that I can never pass.
    - jbaber 3y ago
      For me this happens because of some combination o| - firefox - userAgent spoofing claiming I'm chrome - pihole If I change enough of these, I get through the infinite loop. Does not make me happy.
- victorbjorklund 3y ago
  Cloudflare pages has unlimited free bandwidth so you dont need a limit because traffic is free.
- leerob 3y ago
  (I work at Vercel) We do have DDoS protection, and spend limits. Working on releasing some further improvements to both of these imminently as well. https://vercel.com/blog/introducing-spend-management-realtime-usage-alerts-sms-notifications https://vercel.com/blog/introducing-spend-management-realtim...
Havoc 3y ago
This feels misnamed. It’s not really the serverless part that is the problem but rather inability to limit it eg cloudflare has serverless stuff that you can hard limit
mobilio 3y ago
This story is also good: https://news.ycombinator.com/item?id=25372336 https://news.ycombinator.com/item?id=25372336
rpigab 3y ago
I feel like cloud vendors overcharging for things they have the right to charge you are like a car mechanic that tells you you have to pay him a billion dollar for a routine maintenance check of your car because when he was fixing some small item, it broke and he had to get another one and put in in place, but then it broke for the same reason (that is not the mechanic's fault or the item constructor's), so he kept doing that on an infinite loop but after 2 weeks of him doing that without sending you any info, you went to the garage to check on him and told him to stop, so he stops and bills you for 999999999 items that are broken. As a small user and not a large organisation, I'll never be able to read the full documentation on throttling every nook and cranny, I want to set a fixed spending limit that if reached, will kill the whole thing, even lose data if it must. But they won't do that because it hurts their bottom line if people are careful with budgets, and sometimes it's tricky to compute billing ahead of time.
- throwaway11460 3y ago
  Well there are differences. The mechanic from your example caused the situation. Netlify didn't and it might be hard to distinguish between unwanted traffic and a SaaS having overnight success. A more accurate allegory would be if you told the mechanic to do whatever anyone who knows the registration plate asked, and he did so.
pachico 3y ago
I think the biggest Cloud horror, probably not serverless, is that we have accepted to pay for inter-AZs traffic just in case the Cloud provider fails. In other words, they charge (and a lot) for a mitigation for the problems they might have.
raybb 3y ago
The creator of the blog is Andres who is also building coolify[0], a heroku/netflify self-hosted alternative. I wanted to give a shoutout to him and his product. I've been running it for a few months and love it. Really was has been kinda the secret sauce of ease of use for me to start self-hosting things like changedetector, jdownloader, vaultwarden, etc. It also has a pretty nicely growing community where people are contributing new templates (I added one for Syncthing) and helping each other debug. The only issue I've had with it is things kinda fall over if you run out of disk space, which happened when I was running on an instance with just 10GB storage. So a little better alerting or prevention around that would be great but otherwise it has been pretty solid. https://github.com/coollabsio/coolify https://github.com/coollabsio/coolify
Xophmeister 3y ago
Presumably the costs for a platform are passed on to the platform's customers from that platform's cloud provider (with profit). When a spike happens due to, say, a DDOS, does the platform first have to negotiate with the cloud provider to waive the cost? Presumably the costs for a cloud provider are passed on to the cloud provider's customers from that cloud provider's operating costs (with profit). When a spike happens due to, say, a DDOS, does the cloud provider first have to negotiate with its downstream suppliers (e.g., energy, network peering, etc.) to waive the cost? ... How far down the chain does this go? Or maybe insurance steps in at some point -- or every point -- to break the chain?
- bauruine 3y ago
  It's all just lost profit. It doesn't cost AWS more than it does you copying 190TB to your NAS over LAN.
- brodouevencode 3y ago
  Not necessarily - there are some costs we just eat. However in the event of outside actors having a reputable CDN in place to mitigate these things is worth your time and money.
FerretFred 3y ago
Silly question, but while you're still at the making (expensive) mistakes stage, why not set up a cheap on-prem box that can be switched off/trashed with ease? When you're satisfied that you've fixed all the obvious weaknesses, move the software to The Cloud. Or am I missing the point?
0dayz 3y ago
These are just cost nightmares, what I love (hate) is when said serverless gives you the any kind of bizarre errors and some solutions require you to do some arcane "saw it on stackoverflow sub-comment". Oh and it's not like you can jump into the console to debug. I gave up after not even vercel could figure it out and merely told me that I should replace said package.
fancythat 3y ago
The thing I don't understand is why people go for cloud route for the simple sites, when there are options like Github static hosting or self hosting using secure tunnels from Cloudflare or Ngrok? The risk presented by these horrors stories greatly outweigh possible complications from setting something quickly on your own.
jarym 3y ago
Don't Netlify/Vercel run on top of the likes of AWS/GCP? That means they need to markup on those already inflated hyperscaler markups for data transfer.
Scubabear68 3y ago
I agree with the consensus here that the cost issues are a cloud problem, not serverless. Ironically, something like AWS lambda has a default 1,000 simultaneously activated lambda limit, so the platform defaults to throttling. Not that serverless is perfect.
- chedabob 3y ago
  Ye, especially with something like Netlify or Vercel, I'd bet the vast majority of the costs are CDN bandwidth, and it never even touches the functions themselves. Could just as easily be in this situation with AWS CloudFront and an S3 bucket. I have seen AWS Lambda generate large bills even with the 1000 concurrent executions limit, but with the help of upstream services. My memory of it is hazy, but I think over a weekend, a function with some dodgy retry logic managed to generate something like 21TB of CloudWatch logs.
drada 3y ago
[flagged]
maliker 3y ago
I recently signed up for an OpenAI API key, and by default they enforce a quota after which they disable the key, and when you’re ready to serve more requests you increase the quota manually. I’m surprised to learn more companies don’t do this by default to save everyone from these kind of surprise bills. Although now that I think about it, I suspect there are lots of cases where the users just end up paying these.
- dns_snek 3y ago
  The difference is that for OpenAI, inference costs them real money so they have skin in the game. When Netlify overcharges 1000x for bandwidth, they don't actually care if you don't pay because it cost them next to nothing to provide the service.
brodouevencode 3y ago
I've done quite a bit of cloud cost optimization in the past (sometimes referred to as finops). There's a pedantic discussion to be had as to what is serverless and what is PaaS, but that should be saved for another day. The underlying problem I found working with teams is that they all too frequently wanted to use these cloud-native technologies (serverless, etc.) which means thinking in microservices by definition, but usually architected for monoliths. This was accomplished in a few ways: 1) the serverless job did everything for you so in effect it was a monolith 2) they didn't want to use any intermediary storage/cache layer and relied heavily on both sides being up and available (synchronous brittle systems) 3) they went TOO hard in the microservices direction and made things way too complicated 4) their testing suites were half-baked (more the industry/tools fault than the users tbh) and weren't sufficiently air-gapped or mostly incomplete. If you've been around for more than a decade - we have seen this before: AWS specifically sold the concept of "lift-and-shift" to C-suites a dozen years ago and now that MO is so baked in it's going to take a new generation of services with the right selling points to undo it. Lift-and-shift doesn't work because, as stated above, monoliths aren't for the cloud for a sufficiently sophisticated system. It's the reason DHH and others are advocating for "cloud repatriation". And then there are the small cases, comparatively speaking, like the ones listed on this site. These examples are what happens when you give very powerful tooling to people who do not understand it or the implications of it. But too, this is also partly (maybe even mostly) the vendors' fault: "From localhost to https, in seconds. Deploy from Git or your CLI." is the perfect selling point for a DIY-er or someone running a small business. But what comes along with that low barrier to entry is a downside that could be very expensive when the s**t hits the fan.

292 points by franciscop 3y ago

Minor criticism, I get that this is a page that collects links to other pages talking about the horrors of serverless. Sure. Makes perfect sense. I would like to see some small amount of commentary on each page that covers what I am about to read. Not much, but half a paragraph of introduction would be nice.

remram 3y ago

Is there the opposite website, listing cloud services that do have spend limits?

rldjbpin 3y ago

in terms of web hosting, why does hosting using serverless / paas means not using a cdn? it also baffles me that these services cannot tell between legitimate traffic and ddos.

crooked-v 3y ago

Also see the Netlify comment thread linked from one of the Reddit posts (https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-billing-caused-by-ddos/13086 https://answers.netlify.com/t/limit-bandwidth-to-avoid-high-...), where one of their representatives specifically says that if you suffer a DDOS they will do nothing to stop you from getting a bajillion-dollar bandwidth bill, even if you're on the "free" tier.

voidwtf 3y ago
That's my favorite thing about all these services. None of them have an equivalent of a stop-loss. They'll notify you IF you setup an alert, but that's it. Often times the alerts are way after the damage is done, with some billable metrics often lagging in reporting/alerting by hours. The default should be safe with easy options to increase the limits.
- jmathai 3y ago
  Lambda and GCP Cloud Functions do have cost controls. You can specify the maximum number of instances that can be spun up at any given point. I think the UX can make it more friendly to specify the types of traffic and cost you’re expecting but most erroneous spikes are the result of a customer misconfiguration or a legitimate spike in traffic. It’s the former which are problematic for customers. They can request a credit and we (GCP) try and accommodate. It’s not a feature designed for profit. But it also tends to fall below the cut line for prioritization to fix. But it’s in the backlog and doesn’t go unnoticed. (Former PM on GCP Cloud Functions)
  - sroussey 3y ago
    Do users setup cost controls before activating the service? Or is this something they need to know how to do and do on their own?
    - jmathai 3y ago
      Separate from billing, the product-native cost controls which prevent over-consumption is configured at the resource level. gcloud run deploy ..... --max-instances=N
- fragmede 3y ago
  What you really want is prepaid instead of post-paid billing. Charge up the account with $100, and stop working and delete my data if that gets hit, with an auto renew option to recharge some amount every month.
  - voidwtf 3y ago
    That seems a little reductive. I don’t think it’s unreasonable to say, my current spend is ~13k a month. I should be able to set a 20k cap, such that non fixed billable like bandwidth or map tiles get paused while someone determines whether we’d like to proceed at that rate. There is a not so fine line between scaling for a rush/event and a resume generating event.
huijzer 3y ago
I wonder whether they are running out of money which then causes these things. Netlify used to be a very nice and unique service, but then came GitHub, GitLab and Cloudflare Pages who all provide roughly the same services for less money.
hamoodhabibi 3y ago
It used to be we will take your server down. Now its we'll DDOS your serverless website and leave you a 100k bill. I'm not sure how sustainable such business model is. When you owned the server, you could unplug it. Now you have no way of knowing if somebody is going to hit your /api a million times per minute
- andrasbacsai 3y ago
  I also prefer to get a (decaf) coffee, listen to some music while someone DDoS'd my VPS. I prefer to pay few $ / month for my VPS instead of paying thousands and "survive" the DDoS.
  - s9df898r32h 3y ago
    I pay $10/year for my VPS and host a WordPress Woocommerce store on it... It doesn't get much traffic, but it didn't take long to pay for itself either
    - InvOfSmallC 3y ago
      where?
      - s9df898r32h 3y ago
        [dead]
        Filligree 3y ago
        I explicitly block everything in Russia (and China and Hong Kong) from accessing my servers, because otherwise the rate of SSH attacks is so high it actually prevents SSH from working reliably -- the connection pool fills up. Just an FYI; I don't think it's that unusual. Blocking those geoips dramatically reduced my logging volume.
        3y ago
ponector 3y ago
Can you use free tier under a fake name? To drop ridiculous bills. How do they validate the customer?
- throwaway11460 3y ago
  There is no free plan, there is just free limit - you must add a card.
  - s9df898r32h 3y ago
    just use a virtual card and then change the spending limit to $0? or freeze it
    - throwaway11460 3y ago
      That still passes your personal info.
      - s9df898r32h 3y ago
        at least you wont get charged without your knowledge... which in the end, is the first step in getting this non-sense fixed, for a person... of course the government should fix it for everyone. This shit should probably be considered fraud anyways... companies get free passes all the time. My name (s9df898r32h) is my company, I should be able to steal whatever I want /s Why are such large companies so incompetent though...
        remram 3y ago
        You will have to pay it AND a fee for fraud, and possibly their legal fees for tracking you down. You also will have a difficult time arguing that you were in good faith with this kind of shenanigans. Bouncing checks are not a life hack to get stuff for free, they're a crime.
        3y ago
grubbs 3y ago
So what do I do if this happens to me on the free tier? Just never pay the bill and leave? Will they come after me?

LAC-Tech 3y ago

On the one hand I am glad to see sceptisim of serverless gaining mainstream acceptance. On the otherhand, I fear it will just turn into an unreasoned backlash against technology that is often useful if not as often as its early proponents said it would be (See NoSQL).

tisc 3y ago

Costing money is one thing, but not throttling a queue consuming serverless function and effectively DDOSsing yourself is the true horror story working with serverless :)

bigiain 3y ago
That's kinda not "server less" specific though. There's no end of stories of people discovering 10s to thousand in bill run up on AWS (ir Azure or GCP) because they foot gunned themselves and didn't set up alerts and/or automated service suspension using the cloud providers available tools. Part of me thinks you shouldn't be allowed to provision EC2/S3/Lambda/whatever without first having set up a Cloudwatch alert and a rule to stop your service if your bill goes past expected/acceptable limits. Of course Amazon (or Microsoft or Google) could make that _way_ easier than they do. It'd be nice if there was a simple form in your account setting where you could configure it to suspend services if your projected monthly bill passed $limit. I think it's a more generic "horror story of working with other people's computers".

usr1106 3y ago

How is no spending limit a serverless problem? It's a cloud problem. Edit: Well, if you have small cloud servers they might go down under an attack. And if you pay only outgoing like in AWS that's good for you.

simonjgreen 3y ago
It’s an architecture failing. Setting maximums is critical. If you choose a platform that doesn’t support that more fool you. It’s implicitly easier to have a ‘circuit breaker’ in a traditional environment, but still warrants consideration. Load test for success and failure.
happymellon 3y ago
It's not a cloud problem. The cloud is just somebody else's computer. It's either a billing or an architecture problem. Either let me throttle requests to prevent overbilling, or cut me off when I hit a certain value.
Hamuko 3y ago
Because serverless 1) uses pay-as-you-go more often than traditional models 2) scales better to achieve massive usage. If you DDoS my $5/month DigitalOcean VPS, it's not going to increase how much I have to pay DigitalOcean in terms of compute, and most likely the VPS will buckle under load before it can actually rack up massive transfer costs (since those are also pay-as-you-go for DO).
lelanthran 3y ago
> How is no spending limit a serverless problem? > It's a cloud problem. It's a serverless problem because it can be mitigated if using the cloud, but not mitigated if it is serverless. I have a cloud instance. I don't have this problem and never will. If I had serverless instead, I will have this problem.

000ooo000 3y ago

Something feels a bit off about the creator of this site also pushing a competitor. On one hand, they're upfront about it, and on the other.. they're upfront about it. At least it might prompt the companies in question to be a bit less scummy.

andrasbacsai 3y ago
Which competitor do I push exactly? :) (I created serverlesshorrors)
- hurflmurfl 3y ago
  The link to the "competitor" is front and center on the site, so I imagine what you're really trying to say is that this is NOT a competitor for the following reasons: <missing>. Otherwise this seems like a bait question:)
  - d1sxeyes 3y ago
    Andras (who created both Serverless Horrors and Coolify, which I imagine you're describing as the competitor here) is a pretty solid guy from everything I've seen and I've never seen him ask bait questions or engage in bad faith in discussions here or on other platforms. Although he calls Coolify a Netlify 'alternative', it's 100% 'bring-your-own-server', and is not at all serverless. Coolify is basically a (very nice and featureful) fancy frontend to Docker with integration with Git. Coolify is free if you want to install it yourself and don't mind the (minimal) tinkering needed to get it set up and keep it up-to-date, although you can pay a monthly fee to have your own servers managed for you, the main benefits are either to support continued development, or to access priority support. You still need to bring your own server, even if you're paying the management fee. It's not a competitor to Netlify, it's an alternative.
    - hurflmurfl 3y ago
      Just to clarify, I didn't think that Coolify is a competitor. What I was getting at is that we now have a thread with 4 replies: (1) you pitch competing product (2) what product? (3) Coolify (4) it's not competitor because ... I just wanted to point out that comment #2 only seems to prompt question #3, whereas the helpful reply #4 created by yourself obviates the need for #2 and #3.

tutfbhuf 3y ago

I don't think the Netlify case has happend due to serverless architecture. Yes, there are many technical issues with serverless architectures, but getting a huge bill for incoming traffic is independent from a serverless architecture. You could also put your own hardware in a colo and pay for traffic. If your data center provider does not protect you from DDoS and simply bills by the TB, then the same situation would occur. Of course, one could argue that the prices per TB are much cheaper in a colo situation compared to Netlify and therefore a 100k bill is very unlikely to occur, but then I would make this the main point: absurdly high traffic costs and no DDoS mitigation, and wouldn't put it behind a "serverless horrors" title.

adrianN 3y ago
If your stuff scales to infinity your money burn rate also scales to infinity. A single server in a colo can only do so much damage to your wallet before it’s saturated.
- tutfbhuf 3y ago
  Sure, but according to the article, he received the 100k bill because of peak traffic of 60TB of bandwidth per day over several days. If you have a dedicated server in a colo, then a 10GbE connection is very common these days. With a 10GbE connection, you can serve 100TB of traffic in a single day. Hence, if the colo had the same prices as Netlify for traffic, then a single server with a 10GbE connection would be sufficient for such a high bill, no serverless and infinity scaling required, just absurdly high traffic costs.
  - zrn900 3y ago
    Nah.You get a dedicated box at hetzner and get unmetered 1gbit and use as much as you want as long as you dont do mining or piracy etc.
  - albert180 3y ago
    No Colo charges the same insane prices for Traffic as Cloud Providers. Hetzner charges 1€/TB over 20TB for their 10GbE Servers

liampulles 3y ago

I don't think infinite scaling is desirable. You probably want to say "if the value or rate of increase in xyz" metric reaches a certain level, don't scale further, or start scaling down. As a requirement for a serverless deployment, not an option.

merelysounds 3y ago

A kill switch seems a practical and trivial solution. E.g. something toggled when a monthly bill exceeds a configurable amount. I’m guessing many hobby users would set that to $100, end of story. I don’t understand why it’s so unpopular to offer that. I’d imagine the providers would benefit too in the long run.

andrasbacsai 3y ago
It would be practical for the user, but not for the company. Why does this not exist after years (more than a decade since EC2) of cloud computing? Because it is not good for the VC investors.
- catlifeonmars 3y ago
  AWS Lambda has had this knob for quite a few years, it’s called “provisioned capacity”, and it defines an upper bound to function concurrency.
  - merelysounds 3y ago
    While helpful, this is not the same. E.g. a global kill switch could be harder to overlook and easier to manage.
webprofusion 3y ago
You can set a hard budget on most cloud providers, including Azure etc. I actually did this thinking I was just setting a budget alert and that was a new and interesting way to interrupt production.
- merelysounds 3y ago
  > You can set a hard budget on most cloud providers Note that the submission references serverless providers, specifically netlify and vercel. I don’t think these support an easy to configure kill switch - in any case it wasn’t easy to configure in the scenarios listed on the page.

jeroenvlek 3y ago

Seems very inspired by Molly White's page [0] [0] https://www.web3isgoinggreat.com/ https://www.web3isgoinggreat.com/

KaiserPro 3y ago
its in the style of long runnning dailyWTF: https://thedailywtf.com/ https://thedailywtf.com/ There used to be a whole host of specific fuckup like list site, sadly I can't find them anymore.
rnts08 3y ago
This is brilliant, there goes my afternoon. Thank you

shubhamjain 3y ago

Everyone in this thread is talking about how this is not a Serverless problem, but a cloud problem. That's correct, but the point stands. Bandwidth is pure margin and to me, it's a bit scummy to a) charge this much for bandwidth and b) offer no tools to combat attacks that aren't in the customer's control. According to this thread[1], Netlify doesn't even have an option to take a site down for a while if it's under DDoS attack. Netlify has multiple options to solve the problem: billing controls, request throttling, setting bandwidth limit, and waving of overages if it was a genuine DDoS, but it doesn't seem they are interested because that'd be like killing the golden goose. Just look at all the things bunny.net CDN offers [2]. It's disappointing how we acquiesce to explanations of cloud providers that are motivated by nothing other than greed. [1]: https://answers.netlify.com/t/limiting-bandwidth-traffic-to-netlify-on-starter-tier-plan/18163/8 https://answers.netlify.com/t/limiting-bandwidth-traffic-to-... [2]: https://support.bunny.net/hc/en-us/articles/360014190440-Understanding-Pull-Zone-Network-Limits https://support.bunny.net/hc/en-us/articles/360014190440-Und...

simonjgreen 3y ago

Bandwidth charging models from hosts are crazy. They don’t model their upstream costs in any way. Transit providers charge based on 95th% and ports scale based on highest burst not volume passed. I’ve always hated it.

SushiHippie 3y ago

Related submission from yesterday: Netlify just sent me a $104k bill for a simple static site | https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776

bArray 3y ago

> The ddos attack was focused on a file on my site. Yes it’s partly my fault to put a 3.44MB size sound file on my site rather than using a third-party platform like SoundCloud. But still this doesn’t invalidate the point of having protection against such attacks, and limit the spending. This is extremely sad. It's like we are taking steps backward. A 3.44MB should not be an issue and if it is, the answer should not be to host it elsewhere. If there is no other lesson learned here, it's that there is no such thing as 'free'. As others have said, there should be more done to prevent such large losses without some kind of limit. I would also like to point out that VPS's are extremely cheap [1] and extremely easy to manage. They have automatic limits. [1] https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/ https://lowendbox.com/blog/1-vps-1-usd-vps-per-month/

cpursley 3y ago
VPSs being “easy to manage” is a strong option full of assumptions.
- ies7 3y ago
  I remember a comment from this site. Something like: we must study how Amazon and other big cloud's marketing work so great to make this generation developers think that vps or selfhosting are hard. IMO vps was easy before and even easier now to manage.
  - kikimora 3y ago
    Please explain how you rotate ssh keys, store audit logs, backup (and test backup procedures), configure secure network between vpses such that your neighbor cannot eavesdrop.
    - deleted 3y ago
      [deleted]
    - bcaxis 3y ago
      * You don't have to rotate what doesn't get out. Limit ingress to relevant IPs reduces this surface area a lot. * SCP to a system built for storage. Not really essential for many systems - system logs are fine. * Every VPS provider comes with a backup check box. * Tailscale is really simple.

throwawayyy9237 3y ago

Perhaps I'm overreacting, but I had my personal site on Netlify and decided to move it. I don't need anything more than a dumping ground for HTML, and Netlify was "good enough" (I wasn't aware that this issue existed) What brought it closer to home was the characteristics of the recently affect site (same number of daily visits, not popular (a bit niche), etc) where similar to mine. Moving is easy enough, mostly requiring a DNS update, since I prefer to build the HTML locally and then just dumping it somewhere. What struck me as odd, when looking for an alternative is that almost none of the popular solutions Netlify, Vercel, Cloudfare (AFAIK) actually implement a spend limit. This seems like such a basic thing to do...

aetherspawn 3y ago
Cloudflare has DDOS protection which you can set to basically paranoid and once a DDOS starts then everyone will get a captcha. This limits your spend quite effectively.
- Filligree 3y ago
  And also stop me accessing the site entirely. I don't know why, but every site with CF DDoS protection set to paranoid gives me infinite captchas that I can never pass.
  - jbaber 3y ago
    For me this happens because of some combination o| - firefox - userAgent spoofing claiming I'm chrome - pihole If I change enough of these, I get through the infinite loop. Does not make me happy.
victorbjorklund 3y ago
Cloudflare pages has unlimited free bandwidth so you dont need a limit because traffic is free.
leerob 3y ago
(I work at Vercel) We do have DDoS protection, and spend limits. Working on releasing some further improvements to both of these imminently as well. https://vercel.com/blog/introducing-spend-management-realtime-usage-alerts-sms-notifications https://vercel.com/blog/introducing-spend-management-realtim...

Havoc 3y ago

This feels misnamed. It’s not really the serverless part that is the problem but rather inability to limit it eg cloudflare has serverless stuff that you can hard limit

mobilio 3y ago

This story is also good: https://news.ycombinator.com/item?id=25372336 https://news.ycombinator.com/item?id=25372336

rpigab 3y ago

I feel like cloud vendors overcharging for things they have the right to charge you are like a car mechanic that tells you you have to pay him a billion dollar for a routine maintenance check of your car because when he was fixing some small item, it broke and he had to get another one and put in in place, but then it broke for the same reason (that is not the mechanic's fault or the item constructor's), so he kept doing that on an infinite loop but after 2 weeks of him doing that without sending you any info, you went to the garage to check on him and told him to stop, so he stops and bills you for 999999999 items that are broken. As a small user and not a large organisation, I'll never be able to read the full documentation on throttling every nook and cranny, I want to set a fixed spending limit that if reached, will kill the whole thing, even lose data if it must. But they won't do that because it hurts their bottom line if people are careful with budgets, and sometimes it's tricky to compute billing ahead of time.

throwaway11460 3y ago
Well there are differences. The mechanic from your example caused the situation. Netlify didn't and it might be hard to distinguish between unwanted traffic and a SaaS having overnight success. A more accurate allegory would be if you told the mechanic to do whatever anyone who knows the registration plate asked, and he did so.

pachico 3y ago

I think the biggest Cloud horror, probably not serverless, is that we have accepted to pay for inter-AZs traffic just in case the Cloud provider fails. In other words, they charge (and a lot) for a mitigation for the problems they might have.

raybb 3y ago

The creator of the blog is Andres who is also building coolify[0], a heroku/netflify self-hosted alternative. I wanted to give a shoutout to him and his product. I've been running it for a few months and love it. Really was has been kinda the secret sauce of ease of use for me to start self-hosting things like changedetector, jdownloader, vaultwarden, etc. It also has a pretty nicely growing community where people are contributing new templates (I added one for Syncthing) and helping each other debug. The only issue I've had with it is things kinda fall over if you run out of disk space, which happened when I was running on an instance with just 10GB storage. So a little better alerting or prevention around that would be great but otherwise it has been pretty solid. https://github.com/coollabsio/coolify https://github.com/coollabsio/coolify

Xophmeister 3y ago

Presumably the costs for a platform are passed on to the platform's customers from that platform's cloud provider (with profit). When a spike happens due to, say, a DDOS, does the platform first have to negotiate with the cloud provider to waive the cost? Presumably the costs for a cloud provider are passed on to the cloud provider's customers from that cloud provider's operating costs (with profit). When a spike happens due to, say, a DDOS, does the cloud provider first have to negotiate with its downstream suppliers (e.g., energy, network peering, etc.) to waive the cost? ... How far down the chain does this go? Or maybe insurance steps in at some point -- or every point -- to break the chain?

bauruine 3y ago
It's all just lost profit. It doesn't cost AWS more than it does you copying 190TB to your NAS over LAN.
brodouevencode 3y ago
Not necessarily - there are some costs we just eat. However in the event of outside actors having a reputable CDN in place to mitigate these things is worth your time and money.

FerretFred 3y ago

Silly question, but while you're still at the making (expensive) mistakes stage, why not set up a cheap on-prem box that can be switched off/trashed with ease? When you're satisfied that you've fixed all the obvious weaknesses, move the software to The Cloud. Or am I missing the point?

0dayz 3y ago

These are just cost nightmares, what I love (hate) is when said serverless gives you the any kind of bizarre errors and some solutions require you to do some arcane "saw it on stackoverflow sub-comment". Oh and it's not like you can jump into the console to debug. I gave up after not even vercel could figure it out and merely told me that I should replace said package.

fancythat 3y ago

The thing I don't understand is why people go for cloud route for the simple sites, when there are options like Github static hosting or self hosting using secure tunnels from Cloudflare or Ngrok? The risk presented by these horrors stories greatly outweigh possible complications from setting something quickly on your own.

jarym 3y ago

Don't Netlify/Vercel run on top of the likes of AWS/GCP? That means they need to markup on those already inflated hyperscaler markups for data transfer.

Scubabear68 3y ago

I agree with the consensus here that the cost issues are a cloud problem, not serverless. Ironically, something like AWS lambda has a default 1,000 simultaneously activated lambda limit, so the platform defaults to throttling. Not that serverless is perfect.

chedabob 3y ago
Ye, especially with something like Netlify or Vercel, I'd bet the vast majority of the costs are CDN bandwidth, and it never even touches the functions themselves. Could just as easily be in this situation with AWS CloudFront and an S3 bucket. I have seen AWS Lambda generate large bills even with the 1000 concurrent executions limit, but with the help of upstream services. My memory of it is hazy, but I think over a weekend, a function with some dodgy retry logic managed to generate something like 21TB of CloudWatch logs.

drada 3y ago

[flagged]

maliker 3y ago

I recently signed up for an OpenAI API key, and by default they enforce a quota after which they disable the key, and when you’re ready to serve more requests you increase the quota manually. I’m surprised to learn more companies don’t do this by default to save everyone from these kind of surprise bills. Although now that I think about it, I suspect there are lots of cases where the users just end up paying these.

dns_snek 3y ago
The difference is that for OpenAI, inference costs them real money so they have skin in the game. When Netlify overcharges 1000x for bandwidth, they don't actually care if you don't pay because it cost them next to nothing to provide the service.

brodouevencode 3y ago

I've done quite a bit of cloud cost optimization in the past (sometimes referred to as finops). There's a pedantic discussion to be had as to what is serverless and what is PaaS, but that should be saved for another day. The underlying problem I found working with teams is that they all too frequently wanted to use these cloud-native technologies (serverless, etc.) which means thinking in microservices by definition, but usually architected for monoliths. This was accomplished in a few ways: 1) the serverless job did everything for you so in effect it was a monolith 2) they didn't want to use any intermediary storage/cache layer and relied heavily on both sides being up and available (synchronous brittle systems) 3) they went TOO hard in the microservices direction and made things way too complicated 4) their testing suites were half-baked (more the industry/tools fault than the users tbh) and weren't sufficiently air-gapped or mostly incomplete. If you've been around for more than a decade - we have seen this before: AWS specifically sold the concept of "lift-and-shift" to C-suites a dozen years ago and now that MO is so baked in it's going to take a new generation of services with the right selling points to undo it. Lift-and-shift doesn't work because, as stated above, monoliths aren't for the cloud for a sufficiently sophisticated system. It's the reason DHH and others are advocating for "cloud repatriation". And then there are the small cases, comparatively speaking, like the ones listed on this site. These examples are what happens when you give very powerful tooling to people who do not understand it or the implications of it. But too, this is also partly (maybe even mostly) the vendors' fault: "From localhost to https, in seconds. Deploy from Git or your CLI." is the perfect selling point for a DIY-er or someone running a small business. But what comes along with that low barrier to entry is a downside that could be very expensive when the s**t hits the fan.