10 ms·

Serverless: A lesson learned the hard way

204 points by V3loxy 9y ago

Dolores12 9y ago
"a $180 actual cost. I was left with a light headed feeling, it's a lot of money for me" people still play with fire. limit your losses, go with digital ocean or something for 5$/mo flat no matter what.
- tschellenbach 9y ago
  that depends for 5$ it breaks after a certain level of traffic. for many applications its always better to spend $$ instead of things shutting down. (hosting is usually insignificant compared to people, revenue etc)
  - cbhl 9y ago
    If $180 is significant, then this person is probably a student or on a budget, not a business. So they'd probably prefer the downtime.
    - vlucas 9y ago
      Oh come on. If this is a side project, it's likely impacting his personal monthly budget. A sudden, unplanned, $180 expense on a personal budget can EASILY be significant - even if you make six figures.
  - qaq 9y ago
    it's better to have an ability to cap $. With something like DO it'a way easier to control costs compared to AWS.
- godot 9y ago
  His previous blog post actually said that he moved away from the exact $5/mo digital ocean plan you're talking about, to this. I think the author meant to do this less of a "play with fire" way but more of experimenting with new tech way. But yes, I agree that for personal sites, running with your own money, you probably want to stick with something safer like the $5/mo digital ocean box.
  - V3loxy 9y ago
    This is indeed the case, playing around with new tech. I've been a happy customer with DO for years but my own website was the ideal case to try out the whole serverless thing. As my website doesn't get much traffic so it'd cost me next to nothing. I do agree with most of the comments here though and the $5 DO box is the safest choice. I might have been a bit too excited with the new things and failed to think logically :)
  - Piskvorrr 9y ago
    "Oh, look, new tech. It shines! It heats! Ouch, it also burns!" The comparison to fire is quite fitting.
Danihan 9y ago
The idea that $200 matters that much to a European programmer is so bizarre to me. I just spent that on a decent lunch for two in SF...
- toomuchtodo 9y ago
  I live in Florida, and even in Miami $200 is a very nice dinner once or twice a year. $200 for a decent lunch is insane.
  - aetherson 9y ago
    He's being ridiculous. I live and work in SF. Restaurants here are definitely expensive! But my wife and I get "decent lunches" for $60, not $200. A quite nice dinner for date night is probably $120-$150. (And if we were more price-conscious, I'm pretty sure that we could get "decent" lunches and "quite nice" dinners for less than we pay). You can of course spend much more than that if you choose. But $200 is nothing remotely close to the minimum you'd spend for a "decent lunch."
    - Danihan 9y ago
      I didn't say it was the minimum. Just a nice lunch.
    - apetresc 9y ago
      You both remind me of Lucille Bluth. $20 is PLENTY for a "decent" lunch, even in SF... you can be much more than "pretty sure" that you can have dinner for less than $150, too.
- deleted 9y ago
  [deleted]
- F_r_k 9y ago
  That just shows you've got no idea what you're talking about
  - 9y ago
numbsafari 9y ago
"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Best not to leave yourself exposed.
- occultist_throw 9y ago
  > Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. None of the "Cloud providers" offer that. They "claim" that it could impact service - yeah, service of debt that you owe them.
  - deleted 9y ago
    [deleted]
  - mipmap04 9y ago
    Azure has this. When you hit your spending limit, it shuts down your services.
    - occultist_throw 9y ago
      They didn't a year and a half ago. Created a 3k bill for my employer over RemoteApp. Yeah, charge per user, they said. Oh yeah, min 20 users, and we round up - of course in small print at the time. Unless I have hard guarantees, I give "cloud providers" re-loadable cards. Can't take more money than what's on there.
      - concede_pluto 9y ago
        The vendor could still assign the debt to a collection agency or sue; a declined charge does not get you off the hook unless they decide it's not worth pursuing.
le-mark 9y ago
Off topic, but the "serverless" moniker needs to die. I propose "adminless" as in "server I don't have to admin, configure, or patch" as being much more descriptive of whats really going on.
- motoboi 9y ago
  Well, it's not adminless either, as AWS have lots of them keeping the hosts alive and running. But if you can ignore that, you can probably also ignore the fact that your code runs on a server.
- nathancahill 9y ago
  Eh, if I'm deploying a cloud function, the server truly doesn't exist for me. It's more like a Web Worker running in a privileged environment. I'm ok with the name.
  - sidlls 9y ago
    No, it's really a server running your function. You just (are told you) don't have to worry about the server or what it's actually doing.
    - dvfjsdhgfv 9y ago
      > You just (are told you) don't have to worry about the server ...until you receive the $206 bill for the work done by the server.
    - nathancahill 9y ago
      But that works on the VPS level too. We're just talking about higher levels of abstraction on top of hardware. Which one stops being a "server"? I'd argue that once you lose the OS, it's no longer a "server".
rb808 9y ago
The infinitely scalable cloud services have this big problem with surprise bills. In the old days teams often made do with what was available, now it too easy to spin up more resources. I'm surprised people still talk about cloud services as being cheaper esp where developers are free to use what they want.
tomc1985 9y ago
Run+know your infra, none of this is a problem. Serverless is a scam.
- lostcolony 9y ago
  So run my static content only blog on dedicated hardware that I have to administer rather than throw it in an S3 bucket with a Cloudfront on it? No thank you. Qualify your statements.
  - dvfjsdhgfv 9y ago
    You don't really need to administer it, there are plenty of hosting options, not just bare metal. And you can always add Cloudlflare anyway.
    - lostcolony 9y ago
      You're missing my point. "Serverless" just means "Applications built on cloud services rather than server(s) I have to administer". The OP said I should run my own infrastructure. I -could- host my blog by running a web server atop a server I administer, sure. I'd have to take on all the infrastructural tasks of doing that, securing it, ensuring any availability/scalability concerns I may have are taken care of, etc, but I -could- do that. Instead, S3 + Cloudfront (or, sure, any flavor of hosting and edge caching options you care for; I was not implying "Just AWS") means I don't have to worry about any of that. For me, the reduced level of control, increased availability, scalability, and easy "it just works", is worth the tradeoff. As is the pennies per month it costs me given the low utilization and pay-as-you-go model. It's hardly a scam.
      - dvfjsdhgfv 9y ago
        I wouldn't call it a scam. You need a certain level of expertise anyway, otherwise you not only screw up your service, but also, in the case of AWS, lose money. Practically speaking, the amount of work needed to set up a blog on S3 with CloudFlare is not that different from the one needed to deploy, say, a WordPress droplet on DigitalOcean. You just click a few times and it works, you can basically forget it. The only difference is that you are guaranteed not to pay more than $5 or whatever your monthly plan is. Of course scalability is incomparable in both cases, so if it's something that really matters - and matters more than money - of course something like AWS is a better choice.
matchagaucho 9y ago
TLDR: Deployed buggy code with infinite trigger. Hosting costs increased.
- dvfjsdhgfv 9y ago
  The only difference here is that if he deployed it on his own server, he wouldn't have lost money.
cddotdotslash 9y ago
This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.
- eridius 9y ago
  It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?
  - tdb7893 9y ago
    Could you just do serverless where it starts to rate limit one you reach a certain cost? It seems like this is an issue that could be fixed somehow
    - bluGill 9y ago
      Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers get a poor experience and so they are more likely to ask for a refund, or not renew next time.
      - ryandrake 9y ago
        It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs: 1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral 2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging 3. Warn me at $X and stop service/charging immediately
odammit 9y ago
I've built a fair amount of serverless services over the past two years using the Serverless framework, apex and straight API Gateway/Lambda. It's nice not to have to worry about a server, but I feel like there are just as many little things to futz with in serverless architectures especially before "environment" variables existed in Lambda.
benologist 9y ago
There's a lesson in there about how AWS makes a crisp $10m dollar bill for the richest man in the world every day.
- dvfjsdhgfv 9y ago
  I know you're being sarcastic but I feel it's partly true. They announce so many different services, each month something new appears, but this very basic feature - bill capping - asked by users from the very beginning, has never been implemented. It's hard to believe they lack the skill or that it would be much more complicated than the current alert system.
  - sudhirj 9y ago
    According to the folks I've spoken to, they don't do bill capping because they have no way to safely shut down your workloads in any way - they'd prefer to let you know and have you do it. And having that choice is way better than a capping operation destroying your production database or causing downtime for your users.
    - SimonPStevens 9y ago
      I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could be a soft one. i.e. at £x your servers start shutting down, you'll get billed for a few extra minutes over your cap, before things have finished shutting down. Servers are totally capable of being shutdown without destroying databases. Besides, we aren't really talking about production databases at large companies. The people who want caps are devs learning and experimenting. It could come with dislaimers that if you enable a cap and exceed it that your services will go offline unexpectedly, and that may leave databases in inconsistent states. But for a large number of usage scenerios that is a completely acceptable tradeoff. The simple fact is, not having a cap certainly puts me off experimenting with a service due to a fear of a mistake causing a big bill. And developers learning and investigating a technology is what preceeds them recommending that technology to their companies. Last time I looked Azure allows a zero spend cap on free accounts, but you can't change the amount to anything else, and once you remove it you can't switch the cap back on. Thats limited, but it's perfect for a learning environment. If Azure can implement a zero spend cap, there is absolutly no reason that either AWS or Azure can't implement an x spend cap in exactly the same way.
emersonrsantos 9y ago
Serverless still is a marketing tool for cloud providers. It will be useful when it really offers advantages over managing your own servers, especially on the cost and debugging.
thorum 9y ago
I wrote my (small) AWS app so it can run both on AWS and my local machine. Then you can write tests against the higher-level logic like "save this file to S3" and run those tests locally as well. My main challenge with serverless is using Lambda with API Gateway. Lambda has no database connection pooling, so I end up with a ridiculous number of connections to RDS - one for each simultaneous user. I haven't found a solution to this yet, other than not using API Gateway.
- voganmother42 9y ago
  One solution, use external pooler like pgbouncer or mysql-proxy running on a small instance(s)
  - orthecreedence 9y ago
    I'm actually kind of blown away RDS doesn't have pgbouncer installed on the database. That's how we operate...each db server has pgbouncer living right on it. We connected to bouncer, not directly to the DB.
gleenn 9y ago
He obviously has a smaller account so Amazon might be less flexible, but it's worth contacting support and explaining the error. Sometimes they do give your money back in my experience.
- qaq 9y ago
  In my experience flexible starts north of few mil. per month spend so why all the startups are running on AWS is a mystery to me.
tobyjsullivan 9y ago
>This is probably the most stupid thing I ever did. One missing return; ended up costing me $206. No, dear author. Setting up the AWS billing alarm was the smartest thing you ever did. It probably saved you tens of thousands of dollars (or at least the headache associated with fighting Amazon over the bill). Developers make mistakes. It's part of the job. It's not unusual or bad in any way. A bad developer is one who denies that fact and fails to prepare for it. A great developer is one like the author.
- lifeisstillgood 9y ago
  Just wanted to say that last paragraph is one of the simplest descriptions of professionalism in the job I have read - accept and prepare for your own human failings :-)
illuminati1911 9y ago
This has nothing to do with serverless itself but it's rather problem of AWS.
abalone 9y ago
The main issue here is the budget notification emails aren't an adequate mechanism to catch infinite loops. They are too slow and you've already racked up big overages by the time you see it. Idea: use API Gateway to configure a quota to match your budget projections. That will force a hard stop. Would be nice if AWS made this easier.
alsadi 9y ago
If you are big enough and want serverless no-ops yet don't want to pay per burger when you already breed cows then consider kubeless.io
danschumann 9y ago
Fun programming story!
mullen 9y ago
Setup spending alarms for your account. Personally, mine is $5, $10, $15, $20 and so on. At $30, my wife gets paged.
cryptos 9y ago
Hey, at least is was an enterprise cloud scale infinite loop.
sudhirj 9y ago
You did well to have billing alerts enabled. Exactly the same thing happened to be, but I didn't notice for three months - no emails because I'd created an account and domain for a side project. Didn't notice anything on my card because the charge had been declined, but my bank didn't contact me. Finally found out because I knew the local AWS rep (was the relationship manager for the accounts we use at work). Had to apologise and explain the situation in detail to AWS and they forgave the bill. That was tens of thousands of dollars.
- sudhirj 9y ago
  For those talking about this being a 'serverless' problem or not, think they point is that it's a lot easier to shoot yourself in the foot. Great power + responsibility, etc. On regular servers (outside of unbounded autoscaling) mistakes cost a flat rate.
  - 21 9y ago
    Not necessarily. With a regular server you could go viral, your server dies, so you lose also without a bound in lost business/good will/whatever. Also need to take into account the time/effort spent on making the regular server scale, albeit this is also a relatively flat rate.
sah2ed 9y ago
Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would developers pay for such a service to provide automated notifications of service overages for all the major cloud providers? All a developer needs to do immediately after adding a credit card to AWS/Azure/GCP would be to create an IAM role with permission to automatically add and track fine-grained billing alarms and notify via email/sms for any potential billing overages. I think a $60/yr service like this would be useful to protect against future events of bill shock.
- 21 9y ago
  Azure has a feature on their trial account that when you hit your free limit, you can either: a) go into credit (so they will charge you at end of month) b) disable services Maybe AWS/Google also support a hard limit on spending.
- autotune 9y ago
  Looks like it's already been done, at least for AWS, based on a few Google results (not that I've used/tested any of these): https://github.com/Teevity/ice https://github.com/Teevity/ice https://billgist.com/ https://billgist.com/ http://cloudcheckr.com/ http://cloudcheckr.com/
- coldcode 9y ago
  I looked at Lambda (we use AWS a lot at work) and decided to simply stay with a flat rate DO server. I know what it costs, no need to worry.
j45 9y ago
The relatively low barrier to learn just a tiny bit of following a Linode or DO vps hardening & stack setup guide to get an ubuntu server going can go a very long way for development and prototyping environments. It's gotten much, much easier, and is just another form of command line management, similar to the CLI framework tools with your preferred stack. Once that first setup is done, similar to setting up a serverless environment, you are generally restoring backups of your base image and beginning projects from there. It also immensely helps to learn about how to build something to scale that isn't completely reliant on the PaaS layer.
amcleod 9y ago
Not sure if someone has mentioned this already, but you should contact AWS support and ask if they will forgive the bill given it was an error which led to the high costs. I’ve had bills forgiven this way in the past (e.g. forgot to disable an instance that wasn’t really doing anything).
brown9-2 9y ago
You can host a static site on S3 + Cloudfront, so what purpose does Lambda have in this picture?
solidsnack9000 9y ago
Serverless is going to make resource usage a focus in a way it hasn't been for years. The quick feedback, the absence of "all you can eat" pricing and the possibility for savings are all factors in this.
w8rbt 9y ago
Endless loops are now billing issues, just like a DOS.
archii 9y ago
>Keep an eye on your logs, test everything again and again. This is the takeaway quote from this for me.
supertramp_sid 9y ago
I was about to make the same mistake yesterday , but I had written a validation function that would check inside a folder only, and fortunately I did not upload the file in that folder. And the next morning , I read this article... Man I gotta be careful LOL
tapirl 9y ago
The cost by using AWS is hard to foresee. For years, my s3 storage got charged nothing. But some a month, it got charged several dollars. I have migrated all my services to GCE. At least GCE provides free decent quotas for every resource.
fibo 9y ago
I have a 5$ AWS billing alarm.

204 points by V3loxy 9y ago

"a $180 actual cost. I was left with a light headed feeling, it's a lot of money for me" people still play with fire. limit your losses, go with digital ocean or something for 5$/mo flat no matter what.

tschellenbach 9y ago
that depends for 5$ it breaks after a certain level of traffic. for many applications its always better to spend $$ instead of things shutting down. (hosting is usually insignificant compared to people, revenue etc)
- cbhl 9y ago
  If $180 is significant, then this person is probably a student or on a budget, not a business. So they'd probably prefer the downtime.
  - vlucas 9y ago
    Oh come on. If this is a side project, it's likely impacting his personal monthly budget. A sudden, unplanned, $180 expense on a personal budget can EASILY be significant - even if you make six figures.
- qaq 9y ago
  it's better to have an ability to cap $. With something like DO it'a way easier to control costs compared to AWS.
godot 9y ago
His previous blog post actually said that he moved away from the exact $5/mo digital ocean plan you're talking about, to this. I think the author meant to do this less of a "play with fire" way but more of experimenting with new tech way. But yes, I agree that for personal sites, running with your own money, you probably want to stick with something safer like the $5/mo digital ocean box.
- V3loxy 9y ago
  This is indeed the case, playing around with new tech. I've been a happy customer with DO for years but my own website was the ideal case to try out the whole serverless thing. As my website doesn't get much traffic so it'd cost me next to nothing. I do agree with most of the comments here though and the $5 DO box is the safest choice. I might have been a bit too excited with the new things and failed to think logically :)
- Piskvorrr 9y ago
  "Oh, look, new tech. It shines! It heats! Ouch, it also burns!" The comparison to fire is quite fitting.

Danihan 9y ago

The idea that $200 matters that much to a European programmer is so bizarre to me. I just spent that on a decent lunch for two in SF...

toomuchtodo 9y ago
I live in Florida, and even in Miami $200 is a very nice dinner once or twice a year. $200 for a decent lunch is insane.
- aetherson 9y ago
  He's being ridiculous. I live and work in SF. Restaurants here are definitely expensive! But my wife and I get "decent lunches" for $60, not $200. A quite nice dinner for date night is probably $120-$150. (And if we were more price-conscious, I'm pretty sure that we could get "decent" lunches and "quite nice" dinners for less than we pay). You can of course spend much more than that if you choose. But $200 is nothing remotely close to the minimum you'd spend for a "decent lunch."
  - Danihan 9y ago
    I didn't say it was the minimum. Just a nice lunch.
  - apetresc 9y ago
    You both remind me of Lucille Bluth. $20 is PLENTY for a "decent" lunch, even in SF... you can be much more than "pretty sure" that you can have dinner for less than $150, too.
deleted 9y ago
[deleted]
F_r_k 9y ago
That just shows you've got no idea what you're talking about
- 9y ago

numbsafari 9y ago

"The actual cost is now $206 and over $1000 forecasted, it makes me think twice about using pay-per-use services in the future." Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. Also, never use such services without being sure to configure such settings. I like to think of this as a self-inflicted "DDOC" attack: Distributed Denial of Capital. Best not to leave yourself exposed.

occultist_throw 9y ago
> Never use a pay-per-use service that does not include a reasonable "turn off after $X" feature and appropriate warnings. None of the "Cloud providers" offer that. They "claim" that it could impact service - yeah, service of debt that you owe them.
- deleted 9y ago
  [deleted]
- mipmap04 9y ago
  Azure has this. When you hit your spending limit, it shuts down your services.
  - occultist_throw 9y ago
    They didn't a year and a half ago. Created a 3k bill for my employer over RemoteApp. Yeah, charge per user, they said. Oh yeah, min 20 users, and we round up - of course in small print at the time. Unless I have hard guarantees, I give "cloud providers" re-loadable cards. Can't take more money than what's on there.
    - concede_pluto 9y ago
      The vendor could still assign the debt to a collection agency or sue; a declined charge does not get you off the hook unless they decide it's not worth pursuing.

le-mark 9y ago

Off topic, but the "serverless" moniker needs to die. I propose "adminless" as in "server I don't have to admin, configure, or patch" as being much more descriptive of whats really going on.

motoboi 9y ago
Well, it's not adminless either, as AWS have lots of them keeping the hosts alive and running. But if you can ignore that, you can probably also ignore the fact that your code runs on a server.
nathancahill 9y ago
Eh, if I'm deploying a cloud function, the server truly doesn't exist for me. It's more like a Web Worker running in a privileged environment. I'm ok with the name.
- sidlls 9y ago
  No, it's really a server running your function. You just (are told you) don't have to worry about the server or what it's actually doing.
  - dvfjsdhgfv 9y ago
    > You just (are told you) don't have to worry about the server ...until you receive the $206 bill for the work done by the server.
  - nathancahill 9y ago
    But that works on the VPS level too. We're just talking about higher levels of abstraction on top of hardware. Which one stops being a "server"? I'd argue that once you lose the OS, it's no longer a "server".

rb808 9y ago

The infinitely scalable cloud services have this big problem with surprise bills. In the old days teams often made do with what was available, now it too easy to spin up more resources. I'm surprised people still talk about cloud services as being cheaper esp where developers are free to use what they want.

tomc1985 9y ago

Run+know your infra, none of this is a problem. Serverless is a scam.

lostcolony 9y ago
So run my static content only blog on dedicated hardware that I have to administer rather than throw it in an S3 bucket with a Cloudfront on it? No thank you. Qualify your statements.
- dvfjsdhgfv 9y ago
  You don't really need to administer it, there are plenty of hosting options, not just bare metal. And you can always add Cloudlflare anyway.
  - lostcolony 9y ago
    You're missing my point. "Serverless" just means "Applications built on cloud services rather than server(s) I have to administer". The OP said I should run my own infrastructure. I -could- host my blog by running a web server atop a server I administer, sure. I'd have to take on all the infrastructural tasks of doing that, securing it, ensuring any availability/scalability concerns I may have are taken care of, etc, but I -could- do that. Instead, S3 + Cloudfront (or, sure, any flavor of hosting and edge caching options you care for; I was not implying "Just AWS") means I don't have to worry about any of that. For me, the reduced level of control, increased availability, scalability, and easy "it just works", is worth the tradeoff. As is the pennies per month it costs me given the low utilization and pay-as-you-go model. It's hardly a scam.
    - dvfjsdhgfv 9y ago
      I wouldn't call it a scam. You need a certain level of expertise anyway, otherwise you not only screw up your service, but also, in the case of AWS, lose money. Practically speaking, the amount of work needed to set up a blog on S3 with CloudFlare is not that different from the one needed to deploy, say, a WordPress droplet on DigitalOcean. You just click a few times and it works, you can basically forget it. The only difference is that you are guaranteed not to pay more than $5 or whatever your monthly plan is. Of course scalability is incomparable in both cases, so if it's something that really matters - and matters more than money - of course something like AWS is a better choice.

matchagaucho 9y ago

TLDR: Deployed buggy code with infinite trigger. Hosting costs increased.

dvfjsdhgfv 9y ago
The only difference here is that if he deployed it on his own server, he wouldn't have lost money.

cddotdotslash 9y ago

This is not a "Serverless" problem; this is a mistake a developer made that used a pay-per-use system. If I write code that launches EC2 instances and I accidentally set it to launch an instance every second instead of minute because I divided wrong, that's my fault.

eridius 9y ago
It is a serverless issue because if you were using your own server, a mistake like this wouldn't have cost money, it would have just degraded your service (or possibly brought it offline). So I guess the question is, with a mistake like this, is it better to be charged hundreds or thousands of dollars, or to have your service degrade or go offline until you can fix it?
- tdb7893 9y ago
  Could you just do serverless where it starts to rate limit one you reach a certain cost? It seems like this is an issue that could be fixed somehow
  - bluGill 9y ago
    Yes, but that isn't always the right answer. If your system is starting to cost "a lot" is that because of a bug (this case), or is it because your idea just "went viral" and you are not getting tons of paying customers signing up? If it is the latter you do not want any rate limiting, you want everything to scale as fast as possible (I hope there are no bugs on your end). Rate limiting means that your new customers get a poor experience and so they are more likely to ask for a refund, or not renew next time.
    - ryandrake 9y ago
      It's almost as if... they should offer multiple options so customers could choose based on their business/hobby needs: 1. Warn me at $X but don't throttle me for any reason--I'll pay if I go viral 2. Warn me at $X and start throttling until I get to $Y at which point stop service and stop charging 3. Warn me at $X and stop service/charging immediately

odammit 9y ago

I've built a fair amount of serverless services over the past two years using the Serverless framework, apex and straight API Gateway/Lambda. It's nice not to have to worry about a server, but I feel like there are just as many little things to futz with in serverless architectures especially before "environment" variables existed in Lambda.

benologist 9y ago

There's a lesson in there about how AWS makes a crisp $10m dollar bill for the richest man in the world every day.

dvfjsdhgfv 9y ago
I know you're being sarcastic but I feel it's partly true. They announce so many different services, each month something new appears, but this very basic feature - bill capping - asked by users from the very beginning, has never been implemented. It's hard to believe they lack the skill or that it would be much more complicated than the current alert system.
- sudhirj 9y ago
  According to the folks I've spoken to, they don't do bill capping because they have no way to safely shut down your workloads in any way - they'd prefer to let you know and have you do it. And having that choice is way better than a capping operation destroying your production database or causing downtime for your users.
  - SimonPStevens 9y ago
    I'm sorry, I just don't buy that. It doesn't have to be a hard cap, it could be a soft one. i.e. at £x your servers start shutting down, you'll get billed for a few extra minutes over your cap, before things have finished shutting down. Servers are totally capable of being shutdown without destroying databases. Besides, we aren't really talking about production databases at large companies. The people who want caps are devs learning and experimenting. It could come with dislaimers that if you enable a cap and exceed it that your services will go offline unexpectedly, and that may leave databases in inconsistent states. But for a large number of usage scenerios that is a completely acceptable tradeoff. The simple fact is, not having a cap certainly puts me off experimenting with a service due to a fear of a mistake causing a big bill. And developers learning and investigating a technology is what preceeds them recommending that technology to their companies. Last time I looked Azure allows a zero spend cap on free accounts, but you can't change the amount to anything else, and once you remove it you can't switch the cap back on. Thats limited, but it's perfect for a learning environment. If Azure can implement a zero spend cap, there is absolutly no reason that either AWS or Azure can't implement an x spend cap in exactly the same way.

emersonrsantos 9y ago

Serverless still is a marketing tool for cloud providers. It will be useful when it really offers advantages over managing your own servers, especially on the cost and debugging.

thorum 9y ago

I wrote my (small) AWS app so it can run both on AWS and my local machine. Then you can write tests against the higher-level logic like "save this file to S3" and run those tests locally as well. My main challenge with serverless is using Lambda with API Gateway. Lambda has no database connection pooling, so I end up with a ridiculous number of connections to RDS - one for each simultaneous user. I haven't found a solution to this yet, other than not using API Gateway.

voganmother42 9y ago
One solution, use external pooler like pgbouncer or mysql-proxy running on a small instance(s)
- orthecreedence 9y ago
  I'm actually kind of blown away RDS doesn't have pgbouncer installed on the database. That's how we operate...each db server has pgbouncer living right on it. We connected to bouncer, not directly to the DB.

gleenn 9y ago

He obviously has a smaller account so Amazon might be less flexible, but it's worth contacting support and explaining the error. Sometimes they do give your money back in my experience.

qaq 9y ago
In my experience flexible starts north of few mil. per month spend so why all the startups are running on AWS is a mystery to me.

tobyjsullivan 9y ago

>This is probably the most stupid thing I ever did. One missing return; ended up costing me $206. No, dear author. Setting up the AWS billing alarm was the smartest thing you ever did. It probably saved you tens of thousands of dollars (or at least the headache associated with fighting Amazon over the bill). Developers make mistakes. It's part of the job. It's not unusual or bad in any way. A bad developer is one who denies that fact and fails to prepare for it. A great developer is one like the author.

lifeisstillgood 9y ago
Just wanted to say that last paragraph is one of the simplest descriptions of professionalism in the job I have read - accept and prepare for your own human failings :-)

illuminati1911 9y ago

This has nothing to do with serverless itself but it's rather problem of AWS.

abalone 9y ago

The main issue here is the budget notification emails aren't an adequate mechanism to catch infinite loops. They are too slow and you've already racked up big overages by the time you see it. Idea: use API Gateway to configure a quota to match your budget projections. That will force a hard stop. Would be nice if AWS made this easier.

alsadi 9y ago

If you are big enough and want serverless no-ops yet don't want to pay per burger when you already breed cows then consider kubeless.io

danschumann 9y ago

Fun programming story!

mullen 9y ago

Setup spending alarms for your account. Personally, mine is $5, $10, $15, $20 and so on. At $30, my wife gets paged.

cryptos 9y ago

Hey, at least is was an enterprise cloud scale infinite loop.

sudhirj 9y ago

You did well to have billing alerts enabled. Exactly the same thing happened to be, but I didn't notice for three months - no emails because I'd created an account and domain for a side project. Didn't notice anything on my card because the charge had been declined, but my bank didn't contact me. Finally found out because I knew the local AWS rep (was the relationship manager for the accounts we use at work). Had to apologise and explain the situation in detail to AWS and they forgave the bill. That was tens of thousands of dollars.

sudhirj 9y ago
For those talking about this being a 'serverless' problem or not, think they point is that it's a lot easier to shoot yourself in the foot. Great power + responsibility, etc. On regular servers (outside of unbounded autoscaling) mistakes cost a flat rate.
- 21 9y ago
  Not necessarily. With a regular server you could go viral, your server dies, so you lose also without a bound in lost business/good will/whatever. Also need to take into account the time/effort spent on making the regular server scale, albeit this is also a relatively flat rate.

sah2ed 9y ago

Charged at a flat rate like the cost of a Digital Ocean $5/mo instance, would developers pay for such a service to provide automated notifications of service overages for all the major cloud providers? All a developer needs to do immediately after adding a credit card to AWS/Azure/GCP would be to create an IAM role with permission to automatically add and track fine-grained billing alarms and notify via email/sms for any potential billing overages. I think a $60/yr service like this would be useful to protect against future events of bill shock.

21 9y ago
Azure has a feature on their trial account that when you hit your free limit, you can either: a) go into credit (so they will charge you at end of month) b) disable services Maybe AWS/Google also support a hard limit on spending.
autotune 9y ago
Looks like it's already been done, at least for AWS, based on a few Google results (not that I've used/tested any of these): https://github.com/Teevity/ice https://github.com/Teevity/ice https://billgist.com/ https://billgist.com/ http://cloudcheckr.com/ http://cloudcheckr.com/
coldcode 9y ago
I looked at Lambda (we use AWS a lot at work) and decided to simply stay with a flat rate DO server. I know what it costs, no need to worry.

j45 9y ago

The relatively low barrier to learn just a tiny bit of following a Linode or DO vps hardening & stack setup guide to get an ubuntu server going can go a very long way for development and prototyping environments. It's gotten much, much easier, and is just another form of command line management, similar to the CLI framework tools with your preferred stack. Once that first setup is done, similar to setting up a serverless environment, you are generally restoring backups of your base image and beginning projects from there. It also immensely helps to learn about how to build something to scale that isn't completely reliant on the PaaS layer.

amcleod 9y ago

Not sure if someone has mentioned this already, but you should contact AWS support and ask if they will forgive the bill given it was an error which led to the high costs. I’ve had bills forgiven this way in the past (e.g. forgot to disable an instance that wasn’t really doing anything).

brown9-2 9y ago

You can host a static site on S3 + Cloudfront, so what purpose does Lambda have in this picture?

solidsnack9000 9y ago

Serverless is going to make resource usage a focus in a way it hasn't been for years. The quick feedback, the absence of "all you can eat" pricing and the possibility for savings are all factors in this.

w8rbt 9y ago

Endless loops are now billing issues, just like a DOS.

archii 9y ago

>Keep an eye on your logs, test everything again and again. This is the takeaway quote from this for me.

supertramp_sid 9y ago

I was about to make the same mistake yesterday , but I had written a validation function that would check inside a folder only, and fortunately I did not upload the file in that folder. And the next morning , I read this article... Man I gotta be careful LOL

tapirl 9y ago

The cost by using AWS is hard to foresee. For years, my s3 storage got charged nothing. But some a month, it got charged several dollars. I have migrated all my services to GCE. At least GCE provides free decent quotas for every resource.

fibo 9y ago

I have a 5$ AWS billing alarm.