26 ms·

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

274 points by huksley 4y ago

I am using awesome NextJS and serverless-nextjs and deploy my app to CloudFront and Lambda@Edge. I made a mistake and accidentally created a serverless function that called itself. In a recursive loop, with a 30s timeout. I thought I fixed it and deployed the code to the dev environment. I have had an AWS Billing alert (Budgets) set up to prompt me when my monthly budget goes over $300 (my usual bill is $200/month). Imagine the terror when I woke up the next day to see the AWS Billing alert email saying I already owed $1,484! I removed a function and deployed it again in 30 minutes, but it was too late. It has already run for 24 hours, using over 70 million Gb-Second! Only after that I've learned that AWS Billing alerts do not work this way for CloudFront. You get delayed information on charges because they collect them from all regions. On the following day, the bill settled at a shocking $4600. This is more than we have ever spent on AWS all time. CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront). Now, I understand that I should have created CloudWatch alarms, which would alert me when the number of requests exceeds the limit. The problem is, that they need to be set up per region, and I got CloudFront charges from all points of presence. I am a big proponent of the serverless approach. It makes it easy to scale and develop things (e.g., you get PR review version branches for free, both frontend and backend code like Vercel does). But now, I am unsure because such unexpected charges can ruin a side-project or emerging startup. Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that. What is your experience with it? Would you recommend to use to build a new product if you are bootstrapped, 3-person startup?

tiew9Vii 4y ago
I’m not sure if Lambda@Edge supports reserved concurrency like standard Lambdas? For standard Lambdas I always set a low reserved concurrency to prevent these situations, deliberately throttling my max executions. Start with the lowest number and if you frequently see throttling you can then increase it. For a three person startup… I find Lambdas/Serverless/Cloudflare workers fantastic if you are clever with how you architect things, they can really bring the costs down to trivial amounts. However you need to be defensive I.e. reserved concurrency etc and clever normally means lots of extra steps/hoops to work around restrictions or use things ingeniously. It’s fun if you like that, if we are honest with ourselves it’s hard to beat the simplicity of a VPS and a full stack framework like Rails or whatever alternative in your language of preference when just starting out, then break specific functionality out in to other architectures as required.
m00dy 4y ago
It is not easy to catch self calling functions from a static analysis :(
warent 4y ago
Try filing a support ticket with them with this information. I had something vaguely similar happen on GCP and they refunded the full amount
- huksley 4y ago
  Hi, author here! Thanks, already done that, and waiting for it to be reviewed by AWS. Support was very responsive, though.
  - jackson1442 4y ago
    Make sure to set up budgets too! Support will gladly help you with that since it helps prevent this situation in the future :)
salil999 4y ago
AWS support has historically been pretty good about removing these charges. Just be careful next time. I racked up a $8k AWS bill for my university when I was leading a club. A few emails to AWS support and it was all resolved. Although there might've been more leniency since I was a student.
impostervt 4y ago
I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for some relief, and they called me and let me know they were waving all the charges. So, perhaps you too can beg for mercy?
- causi 4y ago
  The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.
  - jenny91 4y ago
    I don't think that's true? I mean sure, you might not legally be liable when someone impersonates you in the real world. But I'm absolutely certain the AWS terms say somewhere that you agree to take care of your creds and are liable for whatever is done with them, etc?
    - brailsafe 4y ago
      Both could be true. A contract can say anything, but it's going to be bound by the legal framework it operates in, and in this case I don't think there's much of a distinction between the digital and real world, except for physical resources not changing hands. Hypothetically, the contract could say Jeff Bezos will come to your house and personally kill you, but there's no consentual murder in most places
    - withinboredom 4y ago
      You can’t sign away your legal rights.
    - 4y ago
orf 4y ago
Open a support ticket with them and they will likely forgive this charge as a one-time gratuity
jermaustin1 4y ago
I rarely use AWS for smaller projects, and prefer to either use Digital Ocean or bare metal from a local data center (well local when I lived in NY). After a surprise bill like this, I would re-evaluate what serverless is actually giving me.
- hericium 4y ago
  Cloud vs. bare metal costs should always be thoroughly calculated. Fragment from an article from current "FreeBSD Journal"[1] > We compared the three-year total cost of ownership of a VPS, such as a DigitalOcean Droplet, against two equivalent leased or purchased bare metal servers. We estimated that the leased option costs about half as much compared to equal resources in the cloud, and owning the servers would cost less than a quarter of the pure cloud options. [1] https://freebsdfoundation.org/our-work/journal/ https://freebsdfoundation.org/our-work/journal/ [1] https://freebsdfoundation.org/wp-content/uploads/2022/06/Journal-MayJune-2022.pdf https://freebsdfoundation.org/wp-content/uploads/2022/06/Jou...
- ugjka 4y ago
  I'm only using AWS for my domain (too lazy to move) and even then I use an external dns manager because aws charges something like 50 cents per a dns record per month. Everything on aws is a clusterfuck designed to suck money out of enterprise businesses
- nickstinemates 4y ago
  DigitalOcean is amazingly simple and I have been a big fan since they launched.
em1sar 4y ago
It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example). For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this? Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it. The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront). Maybe you can argue that you weren't made aware of this but idk... keep us updated
- Waterluvian 4y ago
  What’s the case for not implementing an optional “shut down all my services at $spend and stay shut down until I intervene” ?
  - bombcar 4y ago
    Honestly? Many people would enable it, forget about it, and footgun themselves on the other side. Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.
    - mirker 4y ago
      I’d think if your rate of spending is >$50/hour then that’s nearly-always a bug. The only reason this conversation is taking place is because serverless “infinitely scales”. Autoscaling physical instances has a max limit for similar reasons.
      - hnlmorg 4y ago
        I've experienced plenty of scenarios where costs have quite legitimately spiked. Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.
drdrek 4y ago
AWS may promote the technologies as prototype friendly but at the end of the day its built to be enterprise grade production tool. A company will not even bother with a 4000$ mistake, its just the price of doing business so there is little incentive to address these types of problems. Playing around with AWS for side projects is like using a chainsaw, it can really accelerate your work but if you are going to make a mistake you may lose an arm and a leg :).
- panki27 4y ago
  Chainsaw is the perfect metaphor for this, well done.
- hnlmorg 4y ago
  That's not really fair because while AWS does have a lot of issues, their refund policy isn't one of them. It's usually really easy to present a case for refunding accidental charges.
  - ryandvm 4y ago
    I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest. It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss or in unpredictable ways, but in most cases that would be preferable for these users as opposed to a couple orders of magnitude increase in billing costs. But the cloud providers don't support hard billing limits because they like people fucking up and accidentally running up their bill. After all, it's probably only a small fraction of users that go through all the humiliating rigamarole of unwinding a provisioning mistake. So yeah, good on Amazon for being so generous with the band aids, but maybe they should try a little harder at helping their users not shoot off their toes...
    - riknos314 4y ago
      > It would be fairly simple for them to allow users to set up hard billing limits. former AWS SDE here I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime. The effort to implement this is far from trivial, removes resources from implementing other features that the really large accounts are asking for, and adds complexity with direct availability risks. It's not at all surprising they don't implement this.
ralusek 4y ago
One of my biggest fears. What's to prevent trolls and competitors from just spamming your endpoints in a loop? How do people using pay-per-use infra deal with these problems? I really want to use Lambda for public endpoints but it just scares me.
- thdxr 4y ago
  There are a lot of tools in AWS to deal with this. Api Gateway Rate Limiting, WAF, etc. That said it does take a level of awareness to set these things up. Some of what we try to do at SST is turn on these for you automatically so you're not being punished for not knowing something
- vforvalerio 4y ago
JamesSwift 4y ago
I had an ECS cron job hang one time, so instead of a 30 second compute charge it ended up being almost a full month of continuous runtime. My usual $30 bill was $800, and the estimated charges for the next month based on 2 days of use was $1300. I didn't have a billing alert setup (definitely setup a billing alert!). It could not have been easier to get AWS to remove the charge. A quick email to support with a brief explanation and it was immediately accepted. The hardest part was that they wanted a very specific request for how much I was asking to be refunded. So I had to go back and calculate my average costs per service and compare that to the charged costs. After that it was immediately refunded. They aren't just handing these things out though. They made me read and acknowledge I had read their service agreements and basically swear that I know what happened and it won't happen again. Really painless process overall, all things considered.
upupandup 4y ago
reminds me when I set up an S3 triggered lambda function that also wrote into the same directory. What ensued was millions of files and folders generated recursively. Fortunately, AWS was kind enough to reverse the billing. Had this been Google Cloud, I would've gotten the cold shoulder and a low key threat that if I reverse the transaction I would lose access to my other paid Google products outside GCP :/
- bstpierre 4y ago
  With S3 you don't even need lambda to do something like this S3 has some setting where you can log activity on a bucket into another bucket But that setting allows you to set the destination bucket to be the same bucket that you're monitoring. So ~30s after something happens on the monitored bucket, S3 writes a log into the same bucket. And then that activity triggers the logging again. So every ~30-60s, forever, there's a little log written into the bucket. It takes a while to add up to something noticeable if your monthly AWS bill is already a few digits long. It's super fun to sift through the bucket a few months later when you're trying to figure out if there's any real data in the bucket or just endless logs.
- henearkr 4y ago
  I've had an horror story with Google Cloud, and they have been very helpful. After I explained my situation (a $5k bill for an inactive side/toy project is extremely painful), and I provided great details about what I think happened to cause it, they wrote off the charges.
deleted 4y ago
[deleted]
phyzome 4y ago
Given that you thought you were doing your due diligence and had set up billing alerts, but their billing alerts are incomplete -- they should be on the hook to give you a one-time pass on that particular failure mode.
lysecret 4y ago
I once made a lot of web requested with lambda in a VPC through a NAT. Cost me similar amounts...
capableweb 4y ago
Off-topic maybe, but I'm very curious as it's hard to find hard numbers: > my usual bill is $200/month How many req/second are you serving? What kind of things are happening? It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk. The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.
- rozenmd 4y ago
  I run 1.2 million uptime checks per week, my total AWS bill was $150/mo before I migrated to permanently running VMs - it's definitely doable without trying too hard.
  - logifail 4y ago
    > my total AWS bill was $150/mo before I migrated to permanently running VMs I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM. I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.
    - rozenmd 4y ago
      Definitely - with startups offering 3x free firecracker VMs (of 256MB RAM) these days, there's almost no reason to start ideas with serverless.
  - huksley 4y ago
    Yes, moving to VM is definitely doable, but now, being a 1 person dev team, it is challenging in maintenance. My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.
jtwebman 4y ago
For small projects why do you need the scale? I feel like once you need the scale serverless is the way more expensive then even managed Kubernetes. I still think serverless is hosting services way to make far more money with the illusion that it is easier when it really isn't. Logging is normally a huge pain. Local dev is usually a huge pain. Managing versions is a pain over just git branches especially over multiple environments. It is a pain to setup different environments and full CI/CD. In then end they might be ok with prototypes but real big systems they are huge pain but that is just my real life experience.
- turtlebits 4y ago
  Serverless isn't just about scale, it's about deploying code without having to touch any infrastructure. The lambda free tier is also very generous (1M free requests per month).
- aelzeiny 4y ago
  To expand on this OP, I've done the AWS-full-stack approach in a mid-sized startup. Modern Serverless problems require modern serverless solutions. That ecosystem is simply not as developed as "traditional" web-server CI/CD. Here are some things that you will eventually need to optimize for. - After crossing a certain threshold in scaling needs, Lambda costs more than regular EC2 on ELB - Lambda cold-start times can be a deal-breaker when users first visit your website. If you contact AWS they will tell you to setup a simple cron job that keep lambdas "warm". But AWS provides no visibility in what's warm or cold, or which endpoints link to which lambdas. - Dealing with Cloudwatch logs of various lambda runs (IMHO) is objectively a bad dev experience. Query insights is getting better, but is still a pain to work with. - To reduce deployment and development times, you'll eventually want to deep-dive into lambda layers. Modern problems modern solutions. - One lambda calling and awaiting another lambda is not a supported first-class use-case. There's no API that allows you to get the status of a lambda run. There's a hack around this where you use AWS Step-Functions. Modern problems modern solutions. We're still on AWS full-stack "serverless" for our webserver and realtime stream processor. At the time I didn't know what I was getting my company into. I wish I just made a Flask webserver instead.
- sitzkrieg 4y ago
mrozbarry 4y ago
At work, my development team is contracted with a company that uses AWS, and for better or worse, we have also become the devops team. We have been burned by AWS before, and we have a rule of thumb: if you are deploying new functionality/service communication, after deploy, monitor for 10-15 minutes, with a wide enough window to see if there is a noticable/unexpected change from before the deploy. It always feels like wasted/burned time, but better to waste time than money. AWS is good about reversing accidental charges, though, but life is always easier if you don't even have to contact support.
- ctvo 4y ago
  > if you are deploying new functionality/service communication, after deploy, monitor for 10-15 minutes, with a wide enough window to see if there is a noticable/unexpected change from before the deploy. It always feels like wasted/burned time, but better to waste time than money. ... have you considered automating this? Alarms are pretty straight forward across all cloud platforms. Since you're using AWS: CloudWatch has anomaly detection. I haven't used it personally but perhaps it's worthwhile to look into: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
  - mrozbarry 4y ago
    We have alarms, but things like cloudwatch can be up to 5 minutes before an alarm goes off.
sleepdreamy 4y ago
This is the exact reason I fear AWS. How should I go about learning it when mistakes like this can basically ruin me for the next few months? Not sure if there are safe resources or Free Sandboxes somewhere
- HWR_14 4y ago
  AWS gives you free resources to start with. They just aren't capped, so you could bankrupt yourself. They also seem to be good about fixing honest mistakes, but it's scary and long.
- fswd 4y ago
  Last weekend I went on a code marathon doing a bunch of CDK AWS stuff. Soon as I saw this headline, I was completely shocked and dropped everything to check my usage. Luckily I had 'cdk destroy' the big projects, but I had dozens and dozens of lambda and S3 buckets it failed to clean up. I spent probably 1 hour clicking through the web interface deleting them in fear of what happen to OP I think I'll just stick to LocalStack and VPS.... actually my datacenter friend said I could bring up my deep learning station to this DC might just abandon AWS idea.
f0e4c2f7 4y ago
If you contact support via your Amazon account and explain your error they will often remove some (but usually not all) of the bill. Sorry that happened, always one of the scarier parts of using AWS. This sounds like an especially tricky one with the standard billing alerts not even catching it.
- huksley 4y ago
  Thank you! I already contacted them, my past experience with AWS was pleasant, it is just this CloudFront delayed billing should be better clarified in the docs I suppose.
tlarkworthy 4y ago
I used to work for Firebase, this is a common problem. For my own developer focussed startup I have prevented functions from calling each other to an unbounded depth, exactly so this footgun is removed. The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1]. [1] https://observablehq.com/@endpointservices/webcode-docs#options https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)
jenny91 4y ago
I want to offer two counterpoints to common sentiments here regarding AWS billing. 1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS. 2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!
- pvillano 4y ago
  I just want my hobby server to shut off at if I spend over $XX in a billing period. Is that so insane?
- noasaservice 4y ago
  Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs. And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be. > AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.
  - speedgoose 4y ago
    Azure doesn’t support spending limits on all subscription types though. https://azure.microsoft.com/support/legal/offer-details/ https://azure.microsoft.com/support/legal/offer-details/
    - 4y ago
cnj 4y ago
I stumbled upon this post the other day, maybe you'll find it helpful: https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-shock-your-next-steps/ https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-sho...
zajio1am 4y ago
With all these stories about unexpected bills from cloud providers, are there cloud providers offering services with pre-paid credit instead of ex-post invoicing?
steve_adams_86 4y ago
For the future, I’m not sure if it suits you but I find you can get a far easier, cheaper, and more predictable dev and deploy experience without using something like CloudFront. I once moved a project from AWS to Digitalocean for a small team (AWS was just all they knew, so it was what they used) and I was able to cut the monthly bill down quite a bit. It isn’t that DO is inherently cheaper or better. It’s just dead simple so it’s easy to deploy with only what you actually need, with easy limits and visibility on what gets spun up. In some cases it’s arguably less cost efficient, but it’s really hard to mess up. For the team I was supporting, simply having the visibility and a simple tool was worth a lot in saved time. They previously spent way too much time on AWS, and couldn’t even get the right infrastructure with the time they invested. So, maybe something worth considering at least. Good luck with the bill! You’re certainly not alone (I got an $800 charge for a db I forgot to kill a few years ago).
Beltiras 4y ago
This is partly the reason why I would never use anything besides VMs or baremetal I provision myself. I'd rather have scaling problems I can solve by provisioning more hardware than billing problems because I fudged the setup. Yes, AWS might be good refunding "oopsies" but when trying to bootstrap a business I have better things to do than recover from heart palpitations.
floor_ 4y ago
I feel like once a day there is always a cautionary tale from some absolute dumbass getting reamed by AWS.
camhart 4y ago
You could just as easily have a "standard" setup recursively calling itself, and scaling up multiple instances as a result. This isn't an issue due to serverless. AWS will hopefully issue a refund.
linuxftw 4y ago
Sounds like AWS might refund you based on other responses, but always be prepared in case they don't. Make sure your DNS provider is not the same entity as your hosting provider, so an unresolved bill doesn't result in your domain being held hostage.
iasay 4y ago
What’s even more worrying is the numerous accidents hiding in a $700k a month bill which is our problem.
- mr337 4y ago
  That is a whole new class of problem. Better start parsing those CSV details billing logs for some gems!
rambambram 4y ago
What is this kind of sh1t!? Why do people put their software on a platform where you don't know what it's going to cost upfront? Maybe I don't understand. Maybe there are legitimate use cases for AWS and other 'vague clouds'. And it's not only these kinds of bills you get, but I also heard one even has to pay to get your own stuff off of these platforms. What's wrong with good old webhosting (in whatever shape, size or form)? I understand there are use cases - probably for big big apps - that will benefit from cloud hosting, but that can't be for every project or business. Right? Please enlighten me.
collsni 4y ago
Wow, cloud. The future!! I bet your glad you gave aws that money and didn't just invest it in your own hardware. Serverless is just somebody else's servers.
hoofhearted 4y ago
We're looking at going to a NextJS frontend setup soon and maybe Lambda for SSR... Would Netlify be a good use case to avoid situations like this and keep the hosting bill down? Netlify CMS really has our eye after chatting with the Plausible.io folks.
sirwitti 4y ago
I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on. But then again, I'm not one of the cool kids...
- Keyframe 4y ago
  Yeah, but how will you scale up your multi-dozen-users-at-the-same-time app easily then?
  - salawat 4y ago
    Buy more servers, loadbalance, and don't take the short painful route architecture-wise? It isn't really that hard.
    - valbaca 4y ago
      Can't wait to end my next system design with "just buy more servers. scaling isn't really that hard."
      - grogenaut 4y ago
        I've done exactly this in many meetings. It's a balance between infra costs and dev time optimizing spend. Typically ping pongs back and forth.
    - devmor 4y ago
      There is a point where that becomes exponentially prohibitively expensive. At a former startup where I worked as senior engineer, that was our original approach. Then one weekend we tripled our userbase and horizontally scaling required massive changes in database architecture, sharding solutions, etc. "Just buy more servers and load balance" is the short painful route. Carefully planning out and taking advantage of scalable architecture that can be provided to you less expensively because it too, runs at scale is the hard method. The fact that its easy to shoot yourself in the foot with it doesn't make it the easy route.
rob_c 4y ago
Self-host and stop using the cloud, especially when you clearly don't know what you're doing. > Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that. Honestly why should they? They're very clear about their billing policy and pricing, the only reason they might is good PR/karma from yc posting...
- Komodai 4y ago
  > the only reason they might is good PR/karma from yc posting not really, they've been known to do it before, I personally got a much smaller charge (~$12) waived by just asking nicely.
  - rob_c 4y ago
    That's directly akin to getting the good stuff until you're hooked... Throw away the account, rent a bare metal server or even a self hosted vm if that's too expensive and you have far more freedom to learn things and do it properly.
    - Komodai 4y ago
      I deleted the account right after I got that waived. And I do have a server (or well two, lol)
jbverschoor 4y ago
Good. Now you have learnt a great lesson, and you also realize that production mistakes can cost quite a lot in a short amount of time.
ryanSrich 4y ago
I wish cloud providers had a nuclear option. Like “if my monthly spend hits $X, then just stop everything immediately”. Often these billing issues happen on little hobby projects and things that the owner would clearly be fine taking offline to avoid thousands in fees.
- maccard 4y ago
  > then just stop everything immediately What does stop everything immediately mean for things that aren't compute? Backups and storage, for example.
  - ryanSrich 4y ago
    I was only thinking about this for compute related services, which is where most of these surprise charges come from. I suppose you could have some fine grained rules for other services.
eranation 4y ago
First, based on my experience, they usually are able to waive this, if it's an honest mistake and from a small company. Second, you can apply for credits, you can get at least $1,000 founders credits for AWS Activate, and if you work with any VC, you can get up to $100,000. Lastly, note that Lambda@edge is (much) more expensive than Lambda, the tech stack I personally pick for any new product, is pure old lambda, serverless-express, and good old React / Vue for the frontend with static assets on S3 (with CloudFront and Route53 for custom domain, ACM for TLS). I'm not that familiar with NextJS but I assume it's mostly used to support SSR right? I would look into the tradeoffs of using it with non edge lambda, vs ditching SSR and using Lambda for API calls only.
- huksley 4y ago
  I keep my fingers crossed! We also applied for AWS Activate sometime ago, as a member of YCombinator Startup School. I would really like to deploy to regular Lambdas, they are per region and easier to monitor, but unfortunately there is no support for this in serverless-nextjs right now. Good thing about is it almost automatic - static frontend, SSR and API are deployed appropriately and work perfectly, without too much fuzz. I try to balance between bringing customer value and doing infrastructure work, and now it is clear I should have spend more time on making a better architecture.
daniel-cussen 4y ago
So the whole point of AWS is you can scale up on demand. The point of it is there's no capital expenditures so you can scale up crazy fast. More scale more bills. Bills on demand. Now, there's two sides to the second part of this. The following is a tiny bite of the fruit of the Tree of the Knowledge of Good and Evil. The good is that Amazon has a policy of leniency. So writing them personally and explaining and asking will probably--and for publicly viewable problems I've seen them come through. For me personally too, now I've also entertained a bad impression of them because they sold me some counterfeits, but one of the items I was sure was a counterfeit and which because of this whole crazy thing on the mail and on the phone they ended up forfeiting, turned out to be genuine compared to the genuine item from the maker itself. I could never distinguish the items in any way other than context. But the ill part, and I considered posing this the other way, ill before good, but in this case it's good before ill, is that they actually have to make a profit at some point. Now they make a lot of profit, but this doesn't change that, the basic thing in business is that it's not just granting, it's charging for what you grant. Both. It's not only about survival, it's also about integrity, because if you never charge you starve, meaning in practice you debase your values until you can eat. Which brings me back to the main point, which is this: they used real energy they had to really pay for, and real hardware that depreciated, and you got the code wrong. Now Amazon I've heard doesn't have training wheels, you want training wheels. Like if you play with assembly, you can fuck up your computer, that's $2000, and since here you don't know what you're doing, they will tell you to like replace the logic board, do this whole thing, computer repairmen for sure screw people. Just like riding a bike, you ride with training wheels until you go through the pain of learning to ride for real, and you fall, and you scrape your knees, again and again until you finally ride for real. And you keep falling off your bike indefinitely, just less.
- vforvalerio 4y ago
- doggwalker 4y ago
  Totally agree. I think of about it too compared to the cost of a semester of useless college classes and the guaranteed loss of thousands of dollars. I opened a separate bank account with $2k in it that is the cost of learning cloud computing and for piece of mind if I do something crazy. I am also though not leaving anything running ever when I log out. No way I am ready to run a lambda function in production.
PaywallBuster 4y ago
maybe doing recursive calls on lambda was not the best course of action? I've been in teams making dozens of lambda based apps without issues, lambda itself is the smallest item on the bill. API Gateway is multiple times more expensive than the compute (lambda) fees
dabinat 4y ago
I think every developer has an AWS billing horror story. My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours. I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website. Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k. I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges. I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access. Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.
- cyberpunk 4y ago
  How can two few gb files = 10k usd of data in a few hours? How popular is that vietnamese porn site!!
  - tcmart14 4y ago
    > How popular is that vietnamese porn site!! Getting to the important question here!
  - bastawhiz 4y ago
    Worth noting that if your distribution is set to use every region, Asia-Pacific CF pricing is actually more expensive than raw S3. $0.12 vs 0.09 (for S3) or 0.085 (for North America CF). It's easy to accidentally increase your costs by 33%, since you'd only encounter this with S3 if you put the bucket in an asian region, versus CF where distributions are more hand-wavey about locations.
Havoc 4y ago
Really wish the major clouds had a “I’m experimenting. Kill it if it goes over 1k usd” mode. Not some crappy alerts - actual halt (and delete if necessary) I bet that would increase profits too by encouraging people try experiment more
- lostmsu 4y ago
  According to other threads here Azure has this mode.
  - Havoc 4y ago
    Not to my knowledge. Certain types of accounts that come with credits have that as a hardcap, but its not something that can be enabled as such https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit https://docs.microsoft.com/en-us/azure/cost-management-billi...
mfrye0 4y ago
I started off using Lambda as well and made the same sort of mistake. I can't remember exactly how much my bill was, but it was enough that it would have drained all my savings and effectively kill my startup. AWS was kind enough to write off most of it. We now use Lambda only for simple cron / background tasks, or consuming from Kinesis. We use ECS for everything else. ECS is nice because it's relatively simple compared to K8, but still gives the full benefit of running multiple containers on one box. I wrote a blog post last year about our migration over to ECS and experimenting with various ways to cut costs: https://blog.bigpicture.io/how-we-cut-our-aws-bill-by-70/ https://blog.bigpicture.io/how-we-cut-our-aws-bill-by-70/
- mfrye0 4y ago
  On a side note, I don't know if you've already acquired any AWS credits. If not, Product Hunt Founders Club is a decent deal that will give you $5k in AWS credits. Between that and the no Stripe fees for 1 year, it paid for itself in no time. https://www.producthunt.com/founder-club https://www.producthunt.com/founder-club
  - huksley 4y ago
    Thank you! I recently applied for AWS Activate credits because our startup was a part of YCombinator Startup School recently.
    - rob_c 4y ago
- huksley 4y ago
  Thank your for ECS suggestion. I am definitely considering it, but struggling to choose between ECS, Elastic Beanstalk or EC2. My past experience with ECS was a bit frustrating because I was forced to use CodeCommit to deploy and I didn't liked that. I would prefer to deploy directly from CI, for example from GitHub actions.
  - 4y ago
hatware 4y ago
The cloud is way over-used and over-relied upon. Most companies could run on a cluster of raspberry pis or equivalent single computer before they need to worry about the cloud.
groffee 4y ago
Just RTFM next time.
jedberg 4y ago
This is a well known but poorly publicized issue with Lambda -- that they can get stuck in infinite loops and run up your bill. I advise anyone I work with that if you are calling one lambda from another anywhere in your system, you should generate a request ID with every inbound request and then pass it along with each call as part of the context, and then error out if you see the same request again. The good news is that AWS is aware of this and that their alarms are delayed, and will almost always waive the fees for you if you ask.
danlugo92 4y ago
Yeah, lesson learned. Don't ever use AWS unless you have a limited card (e.g. Privacy.com) and are willing to burn that bridge.
s1k3s 4y ago
While reading through these threads I always get a feeling that everyone's working at YouTube or stuff like that, and they need to serve millions of users per hour. Meanwhile I'm in my corner here with an old-school $20 VPS that does just fine for my 10k users.
- oznog 4y ago
  In 2010 I served a peak a million users per hour with an $80 dedicated server managed at my pleasure with Plesk and my PHP framework. I can respond up to 20k requests per second. Scaling is for worldwide TOP 250 webs or webs designed according to the stupid contemporary fashions.
jokethrowaway 4y ago
Consider it an expensive lesson on not giving your credit card to those scammers. Aws model is incredibly predatory. All the companies I've seen move to AWS ended up spending way more than before for much less gained. And they still needed an ops team, just more expensive than the sysadmin they had before to manage actual servers. I'll stick to VPS
vforvalerio 4y ago
jscheel 4y ago
Speaking from my own experience, with a much, much larger bill: AWS support will most-likely ask you to set up the cloudwatch alarms and billing alerts properly, and to provide a plan to mitigate further issues. Once you've done that, they will be more inclined to reduce the charges.
Borkdude 4y ago
At my wife's company they had a similar issue with a 25k bill. Ouch...
fleddr 4y ago
I very strongly believe AWS needs dead simple dummy-proof budget controls. 15 years and it still does not have it. And with simple I mean this: "when this service hits my daily/weekly/monthly budget cap of x$, STOP IT AUTOMATICALLY" Individuals/freelancers, hobbyists, small startups need this. When I started in this industry and write an endless loop, I'd hang the server. Now you hang your future.
lmarcos 4y ago
> Would you recommend to use to build a new product if you are bootstrapped, 3-person startup? No. Digital ocean droplet + scp is all you need. Hell, even sftp would work if you are less than 10 people.
johnklos 4y ago
Think about it this way: You're a small, three person group. Are you more worried about having a modest colocated server or a small VPS run out of resources because you suddenly became really, really popular, or are you more worried about who'll go without salary for a few weeks because you misconfigured something and got a bill for multiple thousands of dollars? Amazon takes full advantage of the fact that you can get in to deep billing waters very quickly. They can't collect your billing info quickly enough? Bull - and I mean this emphatically - shit. If a company that runs some of the largest datacenters in the world can't communicate metadata in real time, they're either hopelessly inept, or they're failing on purpose to make more $.
fartcannon 4y ago
It's a website. No matter what you call it, the end result is it's a website. Some HTML and a gif or two. Maybe a bit of javascript. Why are you making it so ludicrously complicated?
ydraiggoch57 4y ago
You might want to try CloudMarshal - you can can trial it for free here: https://app.cloudmarshal.co/ https://app.cloudmarshal.co/
ydraiggoch57 4y ago
You might want to try our CloudMarshal App. You can use it on a free trial here: https://app.cloudmarshal.co/ https://app.cloudmarshal.co/
huksley 4y ago
Now that my bill was finalized for June 2022, I finally got an one-time refund from AWS. Here is a follow-up story about this and how I decided to change my cloud stack to avoid this in the future: https://medium.com/@ruslanfg/aws-refunded-after-i-ddosed-myself-with-cloudfront-8cdb967b2a86 https://medium.com/@ruslanfg/aws-refunded-after-i-ddosed-mys... Based on comments here, I decided to go with AWS ECS in the future.

274 points by huksley 4y ago

tiew9Vii 4y ago

I’m not sure if Lambda@Edge supports reserved concurrency like standard Lambdas? For standard Lambdas I always set a low reserved concurrency to prevent these situations, deliberately throttling my max executions. Start with the lowest number and if you frequently see throttling you can then increase it. For a three person startup… I find Lambdas/Serverless/Cloudflare workers fantastic if you are clever with how you architect things, they can really bring the costs down to trivial amounts. However you need to be defensive I.e. reserved concurrency etc and clever normally means lots of extra steps/hoops to work around restrictions or use things ingeniously. It’s fun if you like that, if we are honest with ourselves it’s hard to beat the simplicity of a VPS and a full stack framework like Rails or whatever alternative in your language of preference when just starting out, then break specific functionality out in to other architectures as required.

m00dy 4y ago

It is not easy to catch self calling functions from a static analysis :(

warent 4y ago

Try filing a support ticket with them with this information. I had something vaguely similar happen on GCP and they refunded the full amount

huksley 4y ago
Hi, author here! Thanks, already done that, and waiting for it to be reviewed by AWS. Support was very responsive, though.
- jackson1442 4y ago
  Make sure to set up budgets too! Support will gladly help you with that since it helps prevent this situation in the future :)

salil999 4y ago

AWS support has historically been pretty good about removing these charges. Just be careful next time. I racked up a $8k AWS bill for my university when I was leading a club. A few emails to AWS support and it was all resolved. Although there might've been more leniency since I was a student.

impostervt 4y ago

I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for some relief, and they called me and let me know they were waving all the charges. So, perhaps you too can beg for mercy?

causi 4y ago
The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.
- jenny91 4y ago
  I don't think that's true? I mean sure, you might not legally be liable when someone impersonates you in the real world. But I'm absolutely certain the AWS terms say somewhere that you agree to take care of your creds and are liable for whatever is done with them, etc?
  - brailsafe 4y ago
    Both could be true. A contract can say anything, but it's going to be bound by the legal framework it operates in, and in this case I don't think there's much of a distinction between the digital and real world, except for physical resources not changing hands. Hypothetically, the contract could say Jeff Bezos will come to your house and personally kill you, but there's no consentual murder in most places
  - withinboredom 4y ago
    You can’t sign away your legal rights.
  - 4y ago

orf 4y ago

Open a support ticket with them and they will likely forgive this charge as a one-time gratuity

jermaustin1 4y ago

I rarely use AWS for smaller projects, and prefer to either use Digital Ocean or bare metal from a local data center (well local when I lived in NY). After a surprise bill like this, I would re-evaluate what serverless is actually giving me.

hericium 4y ago
Cloud vs. bare metal costs should always be thoroughly calculated. Fragment from an article from current "FreeBSD Journal"[1] > We compared the three-year total cost of ownership of a VPS, such as a DigitalOcean Droplet, against two equivalent leased or purchased bare metal servers. We estimated that the leased option costs about half as much compared to equal resources in the cloud, and owning the servers would cost less than a quarter of the pure cloud options. [1] https://freebsdfoundation.org/our-work/journal/ https://freebsdfoundation.org/our-work/journal/ [1] https://freebsdfoundation.org/wp-content/uploads/2022/06/Journal-MayJune-2022.pdf https://freebsdfoundation.org/wp-content/uploads/2022/06/Jou...
ugjka 4y ago
I'm only using AWS for my domain (too lazy to move) and even then I use an external dns manager because aws charges something like 50 cents per a dns record per month. Everything on aws is a clusterfuck designed to suck money out of enterprise businesses
nickstinemates 4y ago
DigitalOcean is amazingly simple and I have been a big fan since they launched.

em1sar 4y ago

It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example). For example, it's totally understandable that the alarms can be specified per region, why shouldn't it be like this? Also the global AWS billing $300 alert seems to have worked but you were asleep as far as I understand. If it was a call-out style alert, then you would've noticed in the middle of the night and could've stopped it. The only thing I agree is frustrating is this: > CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront). Maybe you can argue that you weren't made aware of this but idk... keep us updated

Waterluvian 4y ago
What’s the case for not implementing an optional “shut down all my services at $spend and stay shut down until I intervene” ?
- bombcar 4y ago
  Honestly? Many people would enable it, forget about it, and footgun themselves on the other side. Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.
  - mirker 4y ago
    I’d think if your rate of spending is >$50/hour then that’s nearly-always a bug. The only reason this conversation is taking place is because serverless “infinitely scales”. Autoscaling physical instances has a max limit for similar reasons.
    - hnlmorg 4y ago
      I've experienced plenty of scenarios where costs have quite legitimately spiked. Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.

drdrek 4y ago

AWS may promote the technologies as prototype friendly but at the end of the day its built to be enterprise grade production tool. A company will not even bother with a 4000$ mistake, its just the price of doing business so there is little incentive to address these types of problems. Playing around with AWS for side projects is like using a chainsaw, it can really accelerate your work but if you are going to make a mistake you may lose an arm and a leg :).

panki27 4y ago
Chainsaw is the perfect metaphor for this, well done.
hnlmorg 4y ago
That's not really fair because while AWS does have a lot of issues, their refund policy isn't one of them. It's usually really easy to present a case for refunding accidental charges.
- ryandvm 4y ago
  I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest. It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss or in unpredictable ways, but in most cases that would be preferable for these users as opposed to a couple orders of magnitude increase in billing costs. But the cloud providers don't support hard billing limits because they like people fucking up and accidentally running up their bill. After all, it's probably only a small fraction of users that go through all the humiliating rigamarole of unwinding a provisioning mistake. So yeah, good on Amazon for being so generous with the band aids, but maybe they should try a little harder at helping their users not shoot off their toes...
  - riknos314 4y ago
    > It would be fairly simple for them to allow users to set up hard billing limits. former AWS SDE here I don't believe it would be "fairly simple" to build a completely new off switch into 150+ services, likely with multiple integration points in each service. In addition, the mere existence of an off switch introduces new failure points, where failure directly turns into downtime. The effort to implement this is far from trivial, removes resources from implementing other features that the really large accounts are asking for, and adds complexity with direct availability risks. It's not at all surprising they don't implement this.

ralusek 4y ago

One of my biggest fears. What's to prevent trolls and competitors from just spamming your endpoints in a loop? How do people using pay-per-use infra deal with these problems? I really want to use Lambda for public endpoints but it just scares me.

thdxr 4y ago
There are a lot of tools in AWS to deal with this. Api Gateway Rate Limiting, WAF, etc. That said it does take a level of awareness to set these things up. Some of what we try to do at SST is turn on these for you automatically so you're not being punished for not knowing something
vforvalerio 4y ago

JamesSwift 4y ago

I had an ECS cron job hang one time, so instead of a 30 second compute charge it ended up being almost a full month of continuous runtime. My usual $30 bill was $800, and the estimated charges for the next month based on 2 days of use was $1300. I didn't have a billing alert setup (definitely setup a billing alert!). It could not have been easier to get AWS to remove the charge. A quick email to support with a brief explanation and it was immediately accepted. The hardest part was that they wanted a very specific request for how much I was asking to be refunded. So I had to go back and calculate my average costs per service and compare that to the charged costs. After that it was immediately refunded. They aren't just handing these things out though. They made me read and acknowledge I had read their service agreements and basically swear that I know what happened and it won't happen again. Really painless process overall, all things considered.

upupandup 4y ago

reminds me when I set up an S3 triggered lambda function that also wrote into the same directory. What ensued was millions of files and folders generated recursively. Fortunately, AWS was kind enough to reverse the billing. Had this been Google Cloud, I would've gotten the cold shoulder and a low key threat that if I reverse the transaction I would lose access to my other paid Google products outside GCP :/

bstpierre 4y ago
With S3 you don't even need lambda to do something like this S3 has some setting where you can log activity on a bucket into another bucket But that setting allows you to set the destination bucket to be the same bucket that you're monitoring. So ~30s after something happens on the monitored bucket, S3 writes a log into the same bucket. And then that activity triggers the logging again. So every ~30-60s, forever, there's a little log written into the bucket. It takes a while to add up to something noticeable if your monthly AWS bill is already a few digits long. It's super fun to sift through the bucket a few months later when you're trying to figure out if there's any real data in the bucket or just endless logs.
henearkr 4y ago
I've had an horror story with Google Cloud, and they have been very helpful. After I explained my situation (a $5k bill for an inactive side/toy project is extremely painful), and I provided great details about what I think happened to cause it, they wrote off the charges.

deleted 4y ago

[deleted]

phyzome 4y ago

Given that you thought you were doing your due diligence and had set up billing alerts, but their billing alerts are incomplete -- they should be on the hook to give you a one-time pass on that particular failure mode.

lysecret 4y ago

I once made a lot of web requested with lambda in a VPC through a NAT. Cost me similar amounts...

capableweb 4y ago

Off-topic maybe, but I'm very curious as it's hard to find hard numbers: > my usual bill is $200/month How many req/second are you serving? What kind of things are happening? It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month, including traffic and everything. I'm running a database on the same server, and 20% of the requests writes stuff both to disk and to the database, while the rest just reads from DB or disk. The whole setup took around 5 hours to setup on one day, and have been running flawlessly from day one, we haven't had to migrate servers yet after ~6 months of production usage. Probably one day we're gonna have to upgrade the server to a $80/month one, but that's a one-time thing and our revenue easily covers that.

rozenmd 4y ago
I run 1.2 million uptime checks per week, my total AWS bill was $150/mo before I migrated to permanently running VMs - it's definitely doable without trying too hard.
- logifail 4y ago
  > my total AWS bill was $150/mo before I migrated to permanently running VMs I know I keep harping on about this, but [if you have a vague idea what you're doing] you can squeeze an awful lot out of a cheap VM. I'm currently working on a project that we're deliberately prototyping using ultra-cheap VMs. If it takes off we know how to scale it up, if it doesn't the costs stay very very low.
  - rozenmd 4y ago
    Definitely - with startups offering 3x free firecracker VMs (of 256MB RAM) these days, there's almost no reason to start ideas with serverless.
- huksley 4y ago
  Yes, moving to VM is definitely doable, but now, being a 1 person dev team, it is challenging in maintenance. My fear is, being on a vacation, and suddenly this VM dies. It might take too much time to bring it back online, and I might be out of good network coverage.

jtwebman 4y ago

For small projects why do you need the scale? I feel like once you need the scale serverless is the way more expensive then even managed Kubernetes. I still think serverless is hosting services way to make far more money with the illusion that it is easier when it really isn't. Logging is normally a huge pain. Local dev is usually a huge pain. Managing versions is a pain over just git branches especially over multiple environments. It is a pain to setup different environments and full CI/CD. In then end they might be ok with prototypes but real big systems they are huge pain but that is just my real life experience.

turtlebits 4y ago
Serverless isn't just about scale, it's about deploying code without having to touch any infrastructure. The lambda free tier is also very generous (1M free requests per month).
aelzeiny 4y ago
To expand on this OP, I've done the AWS-full-stack approach in a mid-sized startup. Modern Serverless problems require modern serverless solutions. That ecosystem is simply not as developed as "traditional" web-server CI/CD. Here are some things that you will eventually need to optimize for. - After crossing a certain threshold in scaling needs, Lambda costs more than regular EC2 on ELB - Lambda cold-start times can be a deal-breaker when users first visit your website. If you contact AWS they will tell you to setup a simple cron job that keep lambdas "warm". But AWS provides no visibility in what's warm or cold, or which endpoints link to which lambdas. - Dealing with Cloudwatch logs of various lambda runs (IMHO) is objectively a bad dev experience. Query insights is getting better, but is still a pain to work with. - To reduce deployment and development times, you'll eventually want to deep-dive into lambda layers. Modern problems modern solutions. - One lambda calling and awaiting another lambda is not a supported first-class use-case. There's no API that allows you to get the status of a lambda run. There's a hack around this where you use AWS Step-Functions. Modern problems modern solutions. We're still on AWS full-stack "serverless" for our webserver and realtime stream processor. At the time I didn't know what I was getting my company into. I wish I just made a Flask webserver instead.
sitzkrieg 4y ago

mrozbarry 4y ago

At work, my development team is contracted with a company that uses AWS, and for better or worse, we have also become the devops team. We have been burned by AWS before, and we have a rule of thumb: if you are deploying new functionality/service communication, after deploy, monitor for 10-15 minutes, with a wide enough window to see if there is a noticable/unexpected change from before the deploy. It always feels like wasted/burned time, but better to waste time than money. AWS is good about reversing accidental charges, though, but life is always easier if you don't even have to contact support.

ctvo 4y ago
> if you are deploying new functionality/service communication, after deploy, monitor for 10-15 minutes, with a wide enough window to see if there is a noticable/unexpected change from before the deploy. It always feels like wasted/burned time, but better to waste time than money. ... have you considered automating this? Alarms are pretty straight forward across all cloud platforms. Since you're using AWS: CloudWatch has anomaly detection. I haven't used it personally but perhaps it's worthwhile to look into: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
- mrozbarry 4y ago
  We have alarms, but things like cloudwatch can be up to 5 minutes before an alarm goes off.

sleepdreamy 4y ago

This is the exact reason I fear AWS. How should I go about learning it when mistakes like this can basically ruin me for the next few months? Not sure if there are safe resources or Free Sandboxes somewhere

HWR_14 4y ago
AWS gives you free resources to start with. They just aren't capped, so you could bankrupt yourself. They also seem to be good about fixing honest mistakes, but it's scary and long.
fswd 4y ago
Last weekend I went on a code marathon doing a bunch of CDK AWS stuff. Soon as I saw this headline, I was completely shocked and dropped everything to check my usage. Luckily I had 'cdk destroy' the big projects, but I had dozens and dozens of lambda and S3 buckets it failed to clean up. I spent probably 1 hour clicking through the web interface deleting them in fear of what happen to OP I think I'll just stick to LocalStack and VPS.... actually my datacenter friend said I could bring up my deep learning station to this DC might just abandon AWS idea.

f0e4c2f7 4y ago

If you contact support via your Amazon account and explain your error they will often remove some (but usually not all) of the bill. Sorry that happened, always one of the scarier parts of using AWS. This sounds like an especially tricky one with the standard billing alerts not even catching it.

huksley 4y ago
Thank you! I already contacted them, my past experience with AWS was pleasant, it is just this CloudFront delayed billing should be better clarified in the docs I suppose.

tlarkworthy 4y ago

I used to work for Firebase, this is a common problem. For my own developer focussed startup I have prevented functions from calling each other to an unbounded depth, exactly so this footgun is removed. The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1]. [1] https://observablehq.com/@endpointservices/webcode-docs#options https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)

jenny91 4y ago

I want to offer two counterpoints to common sentiments here regarding AWS billing. 1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS. 2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!

pvillano 4y ago
I just want my hobby server to shut off at if I spend over $XX in a billing period. Is that so insane?
noasaservice 4y ago
Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs. And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be. > AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.
- speedgoose 4y ago
  Azure doesn’t support spending limits on all subscription types though. https://azure.microsoft.com/support/legal/offer-details/ https://azure.microsoft.com/support/legal/offer-details/
  - 4y ago

cnj 4y ago

I stumbled upon this post the other day, maybe you'll find it helpful: https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-shock-your-next-steps/ https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-sho...

zajio1am 4y ago

With all these stories about unexpected bills from cloud providers, are there cloud providers offering services with pre-paid credit instead of ex-post invoicing?

steve_adams_86 4y ago

For the future, I’m not sure if it suits you but I find you can get a far easier, cheaper, and more predictable dev and deploy experience without using something like CloudFront. I once moved a project from AWS to Digitalocean for a small team (AWS was just all they knew, so it was what they used) and I was able to cut the monthly bill down quite a bit. It isn’t that DO is inherently cheaper or better. It’s just dead simple so it’s easy to deploy with only what you actually need, with easy limits and visibility on what gets spun up. In some cases it’s arguably less cost efficient, but it’s really hard to mess up. For the team I was supporting, simply having the visibility and a simple tool was worth a lot in saved time. They previously spent way too much time on AWS, and couldn’t even get the right infrastructure with the time they invested. So, maybe something worth considering at least. Good luck with the bill! You’re certainly not alone (I got an $800 charge for a db I forgot to kill a few years ago).

Beltiras 4y ago

This is partly the reason why I would never use anything besides VMs or baremetal I provision myself. I'd rather have scaling problems I can solve by provisioning more hardware than billing problems because I fudged the setup. Yes, AWS might be good refunding "oopsies" but when trying to bootstrap a business I have better things to do than recover from heart palpitations.

floor_ 4y ago

I feel like once a day there is always a cautionary tale from some absolute dumbass getting reamed by AWS.

camhart 4y ago

You could just as easily have a "standard" setup recursively calling itself, and scaling up multiple instances as a result. This isn't an issue due to serverless. AWS will hopefully issue a refund.

linuxftw 4y ago

Sounds like AWS might refund you based on other responses, but always be prepared in case they don't. Make sure your DNS provider is not the same entity as your hosting provider, so an unresolved bill doesn't result in your domain being held hostage.

iasay 4y ago

What’s even more worrying is the numerous accidents hiding in a $700k a month bill which is our problem.

mr337 4y ago
That is a whole new class of problem. Better start parsing those CSV details billing logs for some gems!

rambambram 4y ago

What is this kind of sh1t!? Why do people put their software on a platform where you don't know what it's going to cost upfront? Maybe I don't understand. Maybe there are legitimate use cases for AWS and other 'vague clouds'. And it's not only these kinds of bills you get, but I also heard one even has to pay to get your own stuff off of these platforms. What's wrong with good old webhosting (in whatever shape, size or form)? I understand there are use cases - probably for big big apps - that will benefit from cloud hosting, but that can't be for every project or business. Right? Please enlighten me.

collsni 4y ago

Wow, cloud. The future!! I bet your glad you gave aws that money and didn't just invest it in your own hardware. Serverless is just somebody else's servers.

hoofhearted 4y ago

We're looking at going to a NextJS frontend setup soon and maybe Lambda for SSR... Would Netlify be a good use case to avoid situations like this and keep the hosting bill down? Netlify CMS really has our eye after chatting with the Plausible.io folks.

sirwitti 4y ago

I'm very much on the boring technology side of things with respect to hosting. 40€ / month gets you a very powerful dedicated server that can easily handle millions of requests per day and performs incredibly well and can be managed easily. If you also use containers you even get quite a bit of flexibility and agility. To be honest I don't really understand the sentiment that developers can get away with not knowing basic sysadmin stuff and at the same time have to spend relevant amounts of time, energy and money to get up to speed with cloud solutions, k8s and so on. But then again, I'm not one of the cool kids...

Keyframe 4y ago
Yeah, but how will you scale up your multi-dozen-users-at-the-same-time app easily then?
- salawat 4y ago
  Buy more servers, loadbalance, and don't take the short painful route architecture-wise? It isn't really that hard.
  - valbaca 4y ago
    Can't wait to end my next system design with "just buy more servers. scaling isn't really that hard."
    - grogenaut 4y ago
      I've done exactly this in many meetings. It's a balance between infra costs and dev time optimizing spend. Typically ping pongs back and forth.
  - devmor 4y ago
    There is a point where that becomes exponentially prohibitively expensive. At a former startup where I worked as senior engineer, that was our original approach. Then one weekend we tripled our userbase and horizontally scaling required massive changes in database architecture, sharding solutions, etc. "Just buy more servers and load balance" is the short painful route. Carefully planning out and taking advantage of scalable architecture that can be provided to you less expensively because it too, runs at scale is the hard method. The fact that its easy to shoot yourself in the foot with it doesn't make it the easy route.

rob_c 4y ago

Self-host and stop using the cloud, especially when you clearly don't know what you're doing. > Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that. Honestly why should they? They're very clear about their billing policy and pricing, the only reason they might is good PR/karma from yc posting...

Komodai 4y ago
> the only reason they might is good PR/karma from yc posting not really, they've been known to do it before, I personally got a much smaller charge (~$12) waived by just asking nicely.
- rob_c 4y ago
  That's directly akin to getting the good stuff until you're hooked... Throw away the account, rent a bare metal server or even a self hosted vm if that's too expensive and you have far more freedom to learn things and do it properly.
  - Komodai 4y ago
    I deleted the account right after I got that waived. And I do have a server (or well two, lol)

jbverschoor 4y ago

Good. Now you have learnt a great lesson, and you also realize that production mistakes can cost quite a lot in a short amount of time.

ryanSrich 4y ago

I wish cloud providers had a nuclear option. Like “if my monthly spend hits $X, then just stop everything immediately”. Often these billing issues happen on little hobby projects and things that the owner would clearly be fine taking offline to avoid thousands in fees.

maccard 4y ago
> then just stop everything immediately What does stop everything immediately mean for things that aren't compute? Backups and storage, for example.
- ryanSrich 4y ago
  I was only thinking about this for compute related services, which is where most of these surprise charges come from. I suppose you could have some fine grained rules for other services.

eranation 4y ago

First, based on my experience, they usually are able to waive this, if it's an honest mistake and from a small company. Second, you can apply for credits, you can get at least $1,000 founders credits for AWS Activate, and if you work with any VC, you can get up to $100,000. Lastly, note that Lambda@edge is (much) more expensive than Lambda, the tech stack I personally pick for any new product, is pure old lambda, serverless-express, and good old React / Vue for the frontend with static assets on S3 (with CloudFront and Route53 for custom domain, ACM for TLS). I'm not that familiar with NextJS but I assume it's mostly used to support SSR right? I would look into the tradeoffs of using it with non edge lambda, vs ditching SSR and using Lambda for API calls only.

huksley 4y ago
I keep my fingers crossed! We also applied for AWS Activate sometime ago, as a member of YCombinator Startup School. I would really like to deploy to regular Lambdas, they are per region and easier to monitor, but unfortunately there is no support for this in serverless-nextjs right now. Good thing about is it almost automatic - static frontend, SSR and API are deployed appropriately and work perfectly, without too much fuzz. I try to balance between bringing customer value and doing infrastructure work, and now it is clear I should have spend more time on making a better architecture.

daniel-cussen 4y ago

So the whole point of AWS is you can scale up on demand. The point of it is there's no capital expenditures so you can scale up crazy fast. More scale more bills. Bills on demand. Now, there's two sides to the second part of this. The following is a tiny bite of the fruit of the Tree of the Knowledge of Good and Evil. The good is that Amazon has a policy of leniency. So writing them personally and explaining and asking will probably--and for publicly viewable problems I've seen them come through. For me personally too, now I've also entertained a bad impression of them because they sold me some counterfeits, but one of the items I was sure was a counterfeit and which because of this whole crazy thing on the mail and on the phone they ended up forfeiting, turned out to be genuine compared to the genuine item from the maker itself. I could never distinguish the items in any way other than context. But the ill part, and I considered posing this the other way, ill before good, but in this case it's good before ill, is that they actually have to make a profit at some point. Now they make a lot of profit, but this doesn't change that, the basic thing in business is that it's not just granting, it's charging for what you grant. Both. It's not only about survival, it's also about integrity, because if you never charge you starve, meaning in practice you debase your values until you can eat. Which brings me back to the main point, which is this: they used real energy they had to really pay for, and real hardware that depreciated, and you got the code wrong. Now Amazon I've heard doesn't have training wheels, you want training wheels. Like if you play with assembly, you can fuck up your computer, that's $2000, and since here you don't know what you're doing, they will tell you to like replace the logic board, do this whole thing, computer repairmen for sure screw people. Just like riding a bike, you ride with training wheels until you go through the pain of learning to ride for real, and you fall, and you scrape your knees, again and again until you finally ride for real. And you keep falling off your bike indefinitely, just less.

vforvalerio 4y ago
doggwalker 4y ago
Totally agree. I think of about it too compared to the cost of a semester of useless college classes and the guaranteed loss of thousands of dollars. I opened a separate bank account with $2k in it that is the cost of learning cloud computing and for piece of mind if I do something crazy. I am also though not leaving anything running ever when I log out. No way I am ready to run a lambda function in production.

PaywallBuster 4y ago

maybe doing recursive calls on lambda was not the best course of action? I've been in teams making dozens of lambda based apps without issues, lambda itself is the smallest item on the bill. API Gateway is multiple times more expensive than the compute (lambda) fees

dabinat 4y ago

I think every developer has an AWS billing horror story. My horror story is that my site allows users to upload videos and share them to a limited number of colleagues. When a user requests a video, a CloudFront URL is created that lasts a few hours. I had not thought much about hotlinking because the link only lasts a few hours - what would be the point? Well, those few hours make a big difference when it’s linked on a high traffic website. Turns out someone paid for the cheapest plan ($7) and uploaded two multi-GB files. They hotlinked them on a Vietnamese porn site and ran up charges of almost $10k. I was alerted by Cost Anomaly Detector but it had already run up most of those charges (and the totals CAD listed were much smaller and made it seem like less of a problem, thus delaying my reaction). AWS, to their credit, waived the charges. I had WAF already setup but it wasn’t very helpful for this type of thing. I could only block sites that I already knew about. I ended up going with a Lambda@Edge solution that validates the source site before allowing access. Lessons learned: 1. Customers may abuse things in ways you didn’t predict 2. Cost Anomaly Detector has a delay and only kicks in once charges have accrued. It can save you from an insane bill but won’t save you completely from large bills. 3. AWS can be reasonable about this but the ball is entirely in their court.

cyberpunk 4y ago
How can two few gb files = 10k usd of data in a few hours? How popular is that vietnamese porn site!!
- tcmart14 4y ago
  > How popular is that vietnamese porn site!! Getting to the important question here!
- bastawhiz 4y ago
  Worth noting that if your distribution is set to use every region, Asia-Pacific CF pricing is actually more expensive than raw S3. $0.12 vs 0.09 (for S3) or 0.085 (for North America CF). It's easy to accidentally increase your costs by 33%, since you'd only encounter this with S3 if you put the bucket in an asian region, versus CF where distributions are more hand-wavey about locations.

Havoc 4y ago

Really wish the major clouds had a “I’m experimenting. Kill it if it goes over 1k usd” mode. Not some crappy alerts - actual halt (and delete if necessary) I bet that would increase profits too by encouraging people try experiment more

lostmsu 4y ago
According to other threads here Azure has this mode.
- Havoc 4y ago
  Not to my knowledge. Certain types of accounts that come with credits have that as a hardcap, but its not something that can be enabled as such https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit https://docs.microsoft.com/en-us/azure/cost-management-billi...

mfrye0 4y ago

I started off using Lambda as well and made the same sort of mistake. I can't remember exactly how much my bill was, but it was enough that it would have drained all my savings and effectively kill my startup. AWS was kind enough to write off most of it. We now use Lambda only for simple cron / background tasks, or consuming from Kinesis. We use ECS for everything else. ECS is nice because it's relatively simple compared to K8, but still gives the full benefit of running multiple containers on one box. I wrote a blog post last year about our migration over to ECS and experimenting with various ways to cut costs: https://blog.bigpicture.io/how-we-cut-our-aws-bill-by-70/ https://blog.bigpicture.io/how-we-cut-our-aws-bill-by-70/

mfrye0 4y ago
On a side note, I don't know if you've already acquired any AWS credits. If not, Product Hunt Founders Club is a decent deal that will give you $5k in AWS credits. Between that and the no Stripe fees for 1 year, it paid for itself in no time. https://www.producthunt.com/founder-club https://www.producthunt.com/founder-club
- huksley 4y ago
  Thank you! I recently applied for AWS Activate credits because our startup was a part of YCombinator Startup School recently.
  - rob_c 4y ago
huksley 4y ago
Thank your for ECS suggestion. I am definitely considering it, but struggling to choose between ECS, Elastic Beanstalk or EC2. My past experience with ECS was a bit frustrating because I was forced to use CodeCommit to deploy and I didn't liked that. I would prefer to deploy directly from CI, for example from GitHub actions.
- 4y ago

hatware 4y ago

The cloud is way over-used and over-relied upon. Most companies could run on a cluster of raspberry pis or equivalent single computer before they need to worry about the cloud.

groffee 4y ago

Just RTFM next time.

jedberg 4y ago

This is a well known but poorly publicized issue with Lambda -- that they can get stuck in infinite loops and run up your bill. I advise anyone I work with that if you are calling one lambda from another anywhere in your system, you should generate a request ID with every inbound request and then pass it along with each call as part of the context, and then error out if you see the same request again. The good news is that AWS is aware of this and that their alarms are delayed, and will almost always waive the fees for you if you ask.

danlugo92 4y ago

Yeah, lesson learned. Don't ever use AWS unless you have a limited card (e.g. Privacy.com) and are willing to burn that bridge.

s1k3s 4y ago

While reading through these threads I always get a feeling that everyone's working at YouTube or stuff like that, and they need to serve millions of users per hour. Meanwhile I'm in my corner here with an old-school $20 VPS that does just fine for my 10k users.

oznog 4y ago
In 2010 I served a peak a million users per hour with an $80 dedicated server managed at my pleasure with Plesk and my PHP framework. I can respond up to 20k requests per second. Scaling is for worldwide TOP 250 webs or webs designed according to the stupid contemporary fashions.

jokethrowaway 4y ago

Consider it an expensive lesson on not giving your credit card to those scammers. Aws model is incredibly predatory. All the companies I've seen move to AWS ended up spending way more than before for much less gained. And they still needed an ops team, just more expensive than the sysadmin they had before to manage actual servers. I'll stick to VPS

vforvalerio 4y ago

jscheel 4y ago

Speaking from my own experience, with a much, much larger bill: AWS support will most-likely ask you to set up the cloudwatch alarms and billing alerts properly, and to provide a plan to mitigate further issues. Once you've done that, they will be more inclined to reduce the charges.

Borkdude 4y ago

At my wife's company they had a similar issue with a 25k bill. Ouch...

fleddr 4y ago

I very strongly believe AWS needs dead simple dummy-proof budget controls. 15 years and it still does not have it. And with simple I mean this: "when this service hits my daily/weekly/monthly budget cap of x$, STOP IT AUTOMATICALLY" Individuals/freelancers, hobbyists, small startups need this. When I started in this industry and write an endless loop, I'd hang the server. Now you hang your future.

lmarcos 4y ago

> Would you recommend to use to build a new product if you are bootstrapped, 3-person startup? No. Digital ocean droplet + scp is all you need. Hell, even sftp would work if you are less than 10 people.

johnklos 4y ago

Think about it this way: You're a small, three person group. Are you more worried about having a modest colocated server or a small VPS run out of resources because you suddenly became really, really popular, or are you more worried about who'll go without salary for a few weeks because you misconfigured something and got a bill for multiple thousands of dollars? Amazon takes full advantage of the fact that you can get in to deep billing waters very quickly. They can't collect your billing info quickly enough? Bull - and I mean this emphatically - shit. If a company that runs some of the largest datacenters in the world can't communicate metadata in real time, they're either hopelessly inept, or they're failing on purpose to make more $.

fartcannon 4y ago

It's a website. No matter what you call it, the end result is it's a website. Some HTML and a gif or two. Maybe a bit of javascript. Why are you making it so ludicrously complicated?

ydraiggoch57 4y ago

You might want to try CloudMarshal - you can can trial it for free here: https://app.cloudmarshal.co/ https://app.cloudmarshal.co/

ydraiggoch57 4y ago

You might want to try our CloudMarshal App. You can use it on a free trial here: https://app.cloudmarshal.co/ https://app.cloudmarshal.co/

huksley 4y ago

Now that my bill was finalized for June 2022, I finally got an one-time refund from AWS. Here is a follow-up story about this and how I decided to change my cloud stack to avoid this in the future: https://medium.com/@ruslanfg/aws-refunded-after-i-ddosed-myself-with-cloudfront-8cdb967b2a86 https://medium.com/@ruslanfg/aws-refunded-after-i-ddosed-mys... Based on comments here, I decided to go with AWS ECS in the future.