7 ms·

AWS Lambda Silent Crash – A Platform Failure, Not an Application Bug [pdf]

115 points by nonfamous 1y ago

elzbardico 1y ago
Geez man, couldn't you just try rewriting this small piece of code in another language?
pensatoio 1y ago
While I have no empathy or patience for Amazon’s support model, you’re really asking for it when you write backend code in Node. I get that some big businesses are using JavaScript on the backend, but I’ll bet almost every single one would be better off switching to a type-safe, compiled language.
malfist 1y ago
Maybe share the code so others can see?
- ceejayoz 1y ago
  One of their docs includes a code sample. https://will-o.co/gf4St6hrhY.png https://will-o.co/gf4St6hrhY.png (not my screenshot)
deleted 1y ago
[deleted]
ceejayoz 1y ago
I saw this come up on /r/aws a few days back. This response seemed illuminating: https://www.reddit.com/r/aws/comments/1lxfblk/comment/n2qww9k/ https://www.reddit.com/r/aws/comments/1lxfblk/comment/n2qww9... > Looking at (this section)[https://will-o.co/gf4St6hrhY.png https://will-o.co/gf4St6hrhY.png], it seems like you're trying to queue up an asyncronous task and then return a response. But when a Lambda handler returns a response, that's the end of execution. You can't return an HTTP response and then do more work after that in the same execution; it's just not a capability of the platform. This is documented behavior: "Your function runs until the handler returns a response, exits, or times out". After you return the object with the message, execution will immediately stop even if other tasks had been queued up.
- mcflubbins 1y ago
  If this is the case (it might very well be) I do at least find it odd that no one on AWS' side was able to explain this to them.
  - ceejayoz 1y ago
    Some people have a lot of certainty coupled with a lack of accuracy.
  - semiquaver 1y ago
    Based on the way the document is written, it seems very likely that several people did realize exactly what the misunderstanding was and try to explain it to them.
  - somethingAlex 1y ago
    It may just be such a basic tenant of the platform that no one thought to. You stop getting billed after lambda returns a response so why would you expect computation to continue? This guy expected free lunch.
    - charcircuit 1y ago
      I don't see that on the main AWS Lambda pages. It just says that you pay for what you use. It would make sense that the time billed would be until there is no more code to execute.
      - meepmorp 1y ago
        Yeah, but if you actually read the developer documentation, they explain the execution model pretty extensively.
        everfrustrated 1y ago
        Developers will do anything except read the documentation.
johnduhart 1y ago
Oh wow, a 23-page write up about how the author misunderstood AWS Lambda's execution model [1]. > It emits an event, then immediately returns a response — meaning it always reports success (201), regardless of whether the downstream email handler succeeds or fails. It should be understood that after Lambda returns a response the MicroVM is suspending, interrupting your background HTTP request. There is zero guarantee that the request would succeed. 1: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#runtimes-lifecycle https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-...
- kevin_nisbet 1y ago
  I don’t know about node but a fun abuse of this is background tasks can still sometimes run on a busy lambda as the same process will unsuspend and resuspend the same process. So you can abuse this sometimes for non essential background tasks and to keep things like caches in process. You just cant rely on this since the runtime instead might just cycle out the suspended lambda.
  - johnduhart 1y ago
    Absolutely, I do this at $dayjob to update feature flags and refresh config. Your code just needs to understand that such execution is not guaranteed to happened, and in-flight requests may get interrupted and should be retried.
  - lbreakjai 1y ago
    I wouldn't really call this an abuse. I remember their documentation mentioning it.
  - __turbobrew__ 1y ago
    TiTiler does exactly that. Geospatial rasters are stored in S3, and the lambda retains a cache in memory of loaded data from S3. So if the same lambda execution is used it can return cached data without hitting S3.
- frenchtoast8 1y ago
nemothekid 1y ago
If I'm reading this correctly, then AWS Support dropped the ball here but this isn't a bug in lambda. This is the documented behavior of the lambda runtime. The document is long, and the examples seem contrived, so anyone is free to correct me but as I understand it the lambda didn't crash, after you returned 201, your lambda instance was put to sleep. You aren't guaranteed that any code will remain running after your lambda "ends". I am not sure why AWS Support was unable to communicate this OP. If you are using Lambda with a function URL, you aren't guaranteed that anything after you return your http response remains running. I believe Lambda has some callbacks/signals you can listen to, to ensure your function properly cleans up before the Lambda is frozen, but if you want the lambda to return as fast as possible it seems you are better off having your service publish to an SQS queue instead.
- Everdred2dx 1y ago
  Yep. I've ran into this using Bugsnag for reporting on unhandled exceptions in Python-based lambda functions. The exception handler would get called, but because the library is async by default the HTTP request wouldn't make it out before the runtime was torn down. I sympathize with OP because debugging this was painful, but I'm sorry to say this is sort of just a "you're holding it wrong" situation.
- semiquaver 1y ago
  This document is bizarre. The author is so confidently verbose about something they are clearly misunderstanding, and have been told as much dozens of times. It’s humbling, in a way, to think of times I’ve felt this strongly about something and to consider the possibility I could have been this wrong.
  - tough 1y ago
    stubbornness is a very human thing
    - Zambyte 1y ago
      The juxtaposition between this reply and the sibling > It's pretty clearly written by GPT. is quite funny
sneilan1 1y ago
`This wasn’t a request for clarification. It was a reset — a request for artefacts that have already been delivered, multiple times, across prior case updates, call notes, and debug summaries. The very same package AWS previously reviewed. The same logs they had already quoted. The same failure they had already confirmed.` While the author is probably correct in this being a platform level bug (I have not ran the code to confirm myself though), they should stop being so angry at a corporation doing what corporations do which is run a big bureaucracy and require extraordinary amounts of extra communication. One of job requirements of a principle engineer, often making hundreds of thousands if not millions of dollars a year, is to communicate across levels because it is so hard to do this effectively without being angry or losing your cool. `As a solo engineer, I outpaced AWS’s own diagnostics, rebuilt our deployment stack from scratch, eliminated every confounding factor, and forced AWS to inadvertently reproduce a failure they still refused to acknowledge.`, I do not think that a solo engineer would have the economic clout to get AWS to pay attention to the bug you found. Finding a bug in a system does not imply any ability to fix a bug in a bureaucracy. Plus, given the tone of the article, I have a funny feeling that the author may have rubbed people in AWS the wrong way, preventing more progress from being made on their bug. But, a wise fellow once said, "all progress in this world depends upon the unreasonable man".
- deleted 1y ago
  [deleted]
- ceejayoz 1y ago
  It's not a platform level bug. It's as-designed, as-documented behavior. Return a response and Lambda suspends. They seem to want a fast response with background processing of a potentially slow task. The answer on Lambda (and honestly, elsewhere) is having this function send the payload to a queue, and processing the queue with another function.
  - sneilan1 1y ago
    Yes you’re right. I wrote the parent comment and then someone else also mentioned the lambda is returning before background processing. I missed that one!
donkey_brains 1y ago
I didn’t bother to read this incredibly long write up past the introduction. But other commenters are asserting that the author expected their Lambda function to continue executing an async task after returning an HTTP response, despite numerous attempts by AWS staff to explain why that wouldn’t work. If true, that is _hilarious_. Easily one of the best stories of misguided anger/hubris/Dunning-Kruger I’ve ever heard. That’s like expecting a local function to continue executing after a `return` statement. It’s such a simple, basic principle of the platform they were trying to use. But one thing bothers me…wouldn’t the author have encountered the exact same behavior in Azure? I guess I really will have to read this paper to find out.
- Everdred2dx 1y ago
  They mentioned “rebuilding” in EC2. Probably just moved that over to Azure and no longer are bothering with serverless :)
- pryelluw 1y ago
  I mean, they should just have created another queue, that triggers another lambda that then continues the process after the http response is sent. However, it is still funny when you think that in regular servers you still finish when you respond to the request. My dog keeps getting angry at the door. It won’t open automatically.
- arpinum 1y ago
  Azure functions don't operate in the same way, they are more like AppRunner's architecture. They then layer on the Lambda billing model. It was typical Azure strategy of rushing out a product to show they have all the same features as AWS. It may have improved since I last touched it in 2018.
  - whatthemick 1y ago
    As of 2025 it's still very much the same.
- jiggawatts 1y ago
  > That’s like expecting a local function to continue executing after a `return` statement. Well... they can![1] It's definitely a thing in traditional always-running web server processes, which is all people know these days. Many developers have never written a code for a short-lived process outside of maybe school exercises. Things like CLI tools, background jobs, etc... [1] I get "Later!" printed out... most of the time: https://dotnetfiddle.net/ng93d4 https://dotnetfiddle.net/ng93d4
scarface_74 1y ago
AWS has a clear policy of not providing support for user application code. I’ve never worked at a company that had enterprise level support with a dedicated TAM. It might be different there. However if it is worth anything I did work at AWS for 3.5 years (AWS ProServe). I knew almost immediately what the issue was as soon as I read the article and knew it was documented and expected behavior.
- icedchai 1y ago
  Having worked with Lambda since it was announced over 10 years ago, it was also clear to me. However, I can also understand if one was new to it and read zero documentation about its execution model, they might be confused. I'd expect someone with experience to ask "am I really the first person to encounter this 'obvious bug'?" though.
Analemma_ 1y ago
This post is a good lesson on humility, and why emotionally-heated blog posts might not be a great idea, no matter how satisfying, and you should possibly stick to dry technicalities instead. If OP had just written up their experience without all the sneering and potshots at Amazon support, people would've just gently corrected them. But as it is they look like a tremendous asshole, and certainly not fit for a CTO role, either temperamentally or skill-wise.
- Quarrel 1y ago
  The poor guy has obviously wound himself in knots over this for several months. He went through all this, obviously really bothered by the impersonal Corporation, "solved" his problem by moving to Azure, but then wrote up this massive rant. Sounds like he needs a holiday at the least. The mere idea that reddit has a special Amazon protection team that is the cause of his account suspension is ludicrous. Hopefully this exposure will help him out.
  - lazystar 1y ago
    > Hopefully this exposure will help him out. looks lile he took down his site after some other commentators in this discussion noticed some discrepencies in his resume. specifically his claims of having a PhD and a JD. Con men never learn, they just play the next fool.
    - Quarrel 1y ago
      Wow, missed this. Yikes.
reilly3000 1y ago
What a tremendous waste of resources. As a startup engineer and leader your job is to build value for customers and investors as quickly and effectively as possible, knowing the privilege of their participation with you is fleeting. The time spent in those meetings or writing this vitriolic pdf would have gone a long way towards shipping features, even if you had to survive with a suboptimal approach for a while. Time isn’t cheap and this cost a fortune.
crinkly 1y ago
Whilst this is a combination of user error and shitty support (not unexpected) I’ve seen Amazon directly not understand how their own lambda stuff works before. There is a pretty horrible bug in one of their higher level products that used lambda because they didn’t know that an execution environment can be reused between invocations. Their software assumed entirely incorrectly that it was stateless resulting in us having to, as the first line in the lambda, clean up the previous invocation’s data on the filesystem. After 3 years they haven’t fixed this as far as I am aware. Getting the actual engineers who wrote this on a call was unproductive as well suggesting it would take months to sort out. We built our own in the end.
- x3n0ph3n3 1y ago
  I'm not convinced AWS support didn't tell him exactly what the problem was and he ignored it. There's no reason to believe he's a reliable narrator here.
- PKop 1y ago
  Given that many commentors on here and reddit have already explained his mistakes, yet he has entries in his CV documenting this "bug report" and "publication" of the incident, I don't really see any evidence whatsoever that AWS didn't explain to him his mistaken understanding of how Lambda works or that they provided shitty support.
ianred 1y ago
The title should have been: AWS Lambda Works as Expected – An Application Architectural Bug [pdf] If the designer expects a reliable system that guarantees processing of the request after the return of the status, they cannot rely on a server to not crash the moment the response was sent to the wire. Serverless enforces this well, yet people try to force their view of infinitely reliable infrastructure, which is not based in reality.
PartiallyTyped 1y ago
I work at lambda, and my team would be the one to engage with OP, it most likely would have been me actually to investigate this. The fact that support didn’t engage us seems odd as we have gotten engaged for far more silly stuff. However, in this case you should have awaited for the response from event emitter. The lambda execution model clearly does not allow you to run things past the response as execution is frozen by the sandbox / virtualization software. From there, you’ve most likely caused a timeout when issuing the request.
fancyswimtime 1y ago
can aws sue this guy for product disparagement lol?
UltraSane 1y ago
The only way you can do async in Lambda is to spawn another Lambda function.
zdc1 1y ago
Taking a step back, this is an interesting reality of choosing platforms. Lambda ain't Node.js. Yes, it runs JavaScript, but that doesn't make it Node.js. JavaScript-land is full of frameworks and platforms that let you do less work in exchange for using a particular set of patterns or architecture. This also shows the best and worst of AWS: things are documented, but not always when/where/how you need them. You will still need to learn a few things the hard way when getting your toes wet (so use those "innovation tokens" sparingly).
- bradyd 1y ago
  Lambda's JavaScript runtime is Node.js.
WatchDog 1y ago
This is a simple misunderstanding of the Lambda execution lifecycle, but it's maybe not a great look for AWS support, that no one there was able to explain the problem to this guy. I guess it's possible someone did explain it to him, and David hasn't mentioned that in his diatribe.
- jdeastwood 1y ago
  I actually witnessed almost exactly the same thing happen in my org recently. AWS support was far _too_ nice and actually did escalate to the service team (or claimed to). There was a few days of back and forth which should have been resolved by saying "please read the how lambda works" section of the documentation.
  - the_plus_one 1y ago
    It's been several years at this point, but I used to be a cloud support engineer for AWS (i.e. the first line of defense against support tickets). We were trained to always be kind to the customer, even if we're frustrated with them, so a response like "please read how the lambda works" probably would've gotten me reprimanded. Customers can rate your response from one to five stars, and if I remember correctly, anything less than four required a written explanation to your manager as to what went wrong. That job was a glorified call center; I didn't last very long.
- ceejayoz 1y ago
  > I guess it's possible someone did explain it to him, and David hasn't mentioned that in his diatribe. Not just possible. Quite likely. It seems very clear the answer was "yes, that's how it works on Lambda, and it is not going to change" and this person said "but I want it to work the other way on Lambda".
jakozaur 1y ago
It is possible to talk even to a runtime engineer, but you need to be on their enterprise plan and spend millions per year with AWS. It may sound brutal, but AWS prioritizes based on the account size. Given the AWS Active mention, the author may be living off free AWS credits and has a modest spend afterwards. Not sure if a seven-week investigation is the best use of engineering resources at early startups. Once you hit the limits of PaaS platforms, going EC2/Fargate/Kubernetes seems like a more pragmatic option. AWS Lambda is not for async jobs; once you return HTTP, it's done. Not convinced this is a bug. However, he did manage to get to the front page of Hacker News, and the write-up is quite decent (minus the AWS blame part).
icedchai 1y ago
Perhaps if he added a delay before returning the response he could’ve narrowed down the problem?
jelder 1y ago
Lambda ends when you return. That's the contract. Bro needs a dose of mechanical sympathy and intellectual vulnerability.
bfeynman 1y ago
Some devil's advocate- AWS receives many support requests that initially go to customer support engineers (in call center environments) who act as first defense for pointless simple issues. But you still have ones escalated to actual engineers and technical teams, costing hundreds or thousands of dollars. There are times when its like actual service issues and bugs etc but the number of times you have a long thread between a support engineer who does not even understand what the customer is misunderstanding (because of arrogance like this, and the customers continuing assertion they are not wrong) can go on for days/weeks and is just monumental waste of time when the answer is read the docs.
amir734jj 1y ago
Please correct me if I'm wrong but lambda model is when server responds then there shouldn't be a background task or any compute after that point otherwise how would lambda know when to stop or there wouldn't be a difference between lambda and VM.
hm-nah 1y ago
Then watch…AWS will fix it and not tell you at all. Similarly, I found that Azure Functions were saving secrets in plaintext in the SCM blade even though the Function App itself was using Key Vault References! I throughly documented the issue, reproduced it with fresh infra, filed a bug bounty, etc. Only to have Microsoft say “It’s the intended behavior” and “That’s not applicable for a bug bounty”, etc. Next month I checked the SCM area again…yeah, plaintext secrets were miraculously redacted. That’s the last time I hunt bugs for you MS!
giddily7894 1y ago
[flagged]
- paulc51 1y ago
  Exactly what I thought. His immediate entry in the resume about this whole circus is ludicrous: > Exposed and published a confirmed AWS Lambda runtime failure (Node.js in VPC) — silently crashing workloads post-request. Outdiagnosed L5 AWS engineers using Docker-based rebuilds, AL2023 parity, and deep forensic isolation. Case study cited on Dev.to, Hashnode, and HackerNews. He must also be completely oblivious to external feedback, given his "Case study cited on HackerNews" is all about his methodology and attitude being torn into pieces.
lijok 1y ago
You've spun yourself in knots. Next time ask chatgpt https://chatgpt.com/share/68768710-1e44-800a-9737-29772153b3d4 https://chatgpt.com/share/68768710-1e44-800a-9737-29772153b3.... I would suggest taking this down and hope it silently goes away. If you're a fractional CTO for a startup that spent a week on this, then spent god knows how long to rearchitect for Azure based on this... The startup might find this actionable. --- EDIT: After looking into this more, it seems the `context` passed to the Lambda entrypoint has a `callbackWaitsForEmptyEventLoop` parameter, which defaults to True. https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context..... Did you set this to False somewhere?
paulc51 1y ago
[dead]

115 points by nonfamous 1y ago

Geez man, couldn't you just try rewriting this small piece of code in another language?

While I have no empathy or patience for Amazon’s support model, you’re really asking for it when you write backend code in Node. I get that some big businesses are using JavaScript on the backend, but I’ll bet almost every single one would be better off switching to a type-safe, compiled language.

malfist 1y ago

Maybe share the code so others can see?

ceejayoz 1y ago
One of their docs includes a code sample. https://will-o.co/gf4St6hrhY.png https://will-o.co/gf4St6hrhY.png (not my screenshot)

deleted 1y ago

[deleted]

ceejayoz 1y ago

I saw this come up on /r/aws a few days back. This response seemed illuminating: https://www.reddit.com/r/aws/comments/1lxfblk/comment/n2qww9k/ https://www.reddit.com/r/aws/comments/1lxfblk/comment/n2qww9... > Looking at (this section)[https://will-o.co/gf4St6hrhY.png https://will-o.co/gf4St6hrhY.png], it seems like you're trying to queue up an asyncronous task and then return a response. But when a Lambda handler returns a response, that's the end of execution. You can't return an HTTP response and then do more work after that in the same execution; it's just not a capability of the platform. This is documented behavior: "Your function runs until the handler returns a response, exits, or times out". After you return the object with the message, execution will immediately stop even if other tasks had been queued up.

mcflubbins 1y ago
If this is the case (it might very well be) I do at least find it odd that no one on AWS' side was able to explain this to them.
- ceejayoz 1y ago
  Some people have a lot of certainty coupled with a lack of accuracy.
- semiquaver 1y ago
  Based on the way the document is written, it seems very likely that several people did realize exactly what the misunderstanding was and try to explain it to them.
- somethingAlex 1y ago
  It may just be such a basic tenant of the platform that no one thought to. You stop getting billed after lambda returns a response so why would you expect computation to continue? This guy expected free lunch.
  - charcircuit 1y ago
    I don't see that on the main AWS Lambda pages. It just says that you pay for what you use. It would make sense that the time billed would be until there is no more code to execute.
    - meepmorp 1y ago
      Yeah, but if you actually read the developer documentation, they explain the execution model pretty extensively.
      - everfrustrated 1y ago
        Developers will do anything except read the documentation.

johnduhart 1y ago

Oh wow, a 23-page write up about how the author misunderstood AWS Lambda's execution model [1]. > It emits an event, then immediately returns a response — meaning it always reports success (201), regardless of whether the downstream email handler succeeds or fails. It should be understood that after Lambda returns a response the MicroVM is suspending, interrupting your background HTTP request. There is zero guarantee that the request would succeed. 1: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#runtimes-lifecycle https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-...

kevin_nisbet 1y ago
I don’t know about node but a fun abuse of this is background tasks can still sometimes run on a busy lambda as the same process will unsuspend and resuspend the same process. So you can abuse this sometimes for non essential background tasks and to keep things like caches in process. You just cant rely on this since the runtime instead might just cycle out the suspended lambda.
- johnduhart 1y ago
  Absolutely, I do this at $dayjob to update feature flags and refresh config. Your code just needs to understand that such execution is not guaranteed to happened, and in-flight requests may get interrupted and should be retried.
- lbreakjai 1y ago
  I wouldn't really call this an abuse. I remember their documentation mentioning it.
- __turbobrew__ 1y ago
  TiTiler does exactly that. Geospatial rasters are stored in S3, and the lambda retains a cache in memory of loaded data from S3. So if the same lambda execution is used it can return cached data without hitting S3.
frenchtoast8 1y ago

nemothekid 1y ago

If I'm reading this correctly, then AWS Support dropped the ball here but this isn't a bug in lambda. This is the documented behavior of the lambda runtime. The document is long, and the examples seem contrived, so anyone is free to correct me but as I understand it the lambda didn't crash, after you returned 201, your lambda instance was put to sleep. You aren't guaranteed that any code will remain running after your lambda "ends". I am not sure why AWS Support was unable to communicate this OP. If you are using Lambda with a function URL, you aren't guaranteed that anything after you return your http response remains running. I believe Lambda has some callbacks/signals you can listen to, to ensure your function properly cleans up before the Lambda is frozen, but if you want the lambda to return as fast as possible it seems you are better off having your service publish to an SQS queue instead.

Everdred2dx 1y ago
Yep. I've ran into this using Bugsnag for reporting on unhandled exceptions in Python-based lambda functions. The exception handler would get called, but because the library is async by default the HTTP request wouldn't make it out before the runtime was torn down. I sympathize with OP because debugging this was painful, but I'm sorry to say this is sort of just a "you're holding it wrong" situation.
semiquaver 1y ago
This document is bizarre. The author is so confidently verbose about something they are clearly misunderstanding, and have been told as much dozens of times. It’s humbling, in a way, to think of times I’ve felt this strongly about something and to consider the possibility I could have been this wrong.
- tough 1y ago
  stubbornness is a very human thing
  - Zambyte 1y ago
    The juxtaposition between this reply and the sibling > It's pretty clearly written by GPT. is quite funny

sneilan1 1y ago

`This wasn’t a request for clarification. It was a reset — a request for artefacts that have already been delivered, multiple times, across prior case updates, call notes, and debug summaries. The very same package AWS previously reviewed. The same logs they had already quoted. The same failure they had already confirmed.` While the author is probably correct in this being a platform level bug (I have not ran the code to confirm myself though), they should stop being so angry at a corporation doing what corporations do which is run a big bureaucracy and require extraordinary amounts of extra communication. One of job requirements of a principle engineer, often making hundreds of thousands if not millions of dollars a year, is to communicate across levels because it is so hard to do this effectively without being angry or losing your cool. `As a solo engineer, I outpaced AWS’s own diagnostics, rebuilt our deployment stack from scratch, eliminated every confounding factor, and forced AWS to inadvertently reproduce a failure they still refused to acknowledge.`, I do not think that a solo engineer would have the economic clout to get AWS to pay attention to the bug you found. Finding a bug in a system does not imply any ability to fix a bug in a bureaucracy. Plus, given the tone of the article, I have a funny feeling that the author may have rubbed people in AWS the wrong way, preventing more progress from being made on their bug. But, a wise fellow once said, "all progress in this world depends upon the unreasonable man".

deleted 1y ago
[deleted]
ceejayoz 1y ago
It's not a platform level bug. It's as-designed, as-documented behavior. Return a response and Lambda suspends. They seem to want a fast response with background processing of a potentially slow task. The answer on Lambda (and honestly, elsewhere) is having this function send the payload to a queue, and processing the queue with another function.
- sneilan1 1y ago
  Yes you’re right. I wrote the parent comment and then someone else also mentioned the lambda is returning before background processing. I missed that one!

donkey_brains 1y ago

I didn’t bother to read this incredibly long write up past the introduction. But other commenters are asserting that the author expected their Lambda function to continue executing an async task after returning an HTTP response, despite numerous attempts by AWS staff to explain why that wouldn’t work. If true, that is _hilarious_. Easily one of the best stories of misguided anger/hubris/Dunning-Kruger I’ve ever heard. That’s like expecting a local function to continue executing after a `return` statement. It’s such a simple, basic principle of the platform they were trying to use. But one thing bothers me…wouldn’t the author have encountered the exact same behavior in Azure? I guess I really will have to read this paper to find out.

Everdred2dx 1y ago
They mentioned “rebuilding” in EC2. Probably just moved that over to Azure and no longer are bothering with serverless :)
pryelluw 1y ago
I mean, they should just have created another queue, that triggers another lambda that then continues the process after the http response is sent. However, it is still funny when you think that in regular servers you still finish when you respond to the request. My dog keeps getting angry at the door. It won’t open automatically.
arpinum 1y ago
Azure functions don't operate in the same way, they are more like AppRunner's architecture. They then layer on the Lambda billing model. It was typical Azure strategy of rushing out a product to show they have all the same features as AWS. It may have improved since I last touched it in 2018.
- whatthemick 1y ago
  As of 2025 it's still very much the same.
jiggawatts 1y ago
> That’s like expecting a local function to continue executing after a `return` statement. Well... they can![1] It's definitely a thing in traditional always-running web server processes, which is all people know these days. Many developers have never written a code for a short-lived process outside of maybe school exercises. Things like CLI tools, background jobs, etc... [1] I get "Later!" printed out... most of the time: https://dotnetfiddle.net/ng93d4 https://dotnetfiddle.net/ng93d4

scarface_74 1y ago

AWS has a clear policy of not providing support for user application code. I’ve never worked at a company that had enterprise level support with a dedicated TAM. It might be different there. However if it is worth anything I did work at AWS for 3.5 years (AWS ProServe). I knew almost immediately what the issue was as soon as I read the article and knew it was documented and expected behavior.

icedchai 1y ago
Having worked with Lambda since it was announced over 10 years ago, it was also clear to me. However, I can also understand if one was new to it and read zero documentation about its execution model, they might be confused. I'd expect someone with experience to ask "am I really the first person to encounter this 'obvious bug'?" though.

Analemma_ 1y ago

This post is a good lesson on humility, and why emotionally-heated blog posts might not be a great idea, no matter how satisfying, and you should possibly stick to dry technicalities instead. If OP had just written up their experience without all the sneering and potshots at Amazon support, people would've just gently corrected them. But as it is they look like a tremendous asshole, and certainly not fit for a CTO role, either temperamentally or skill-wise.

Quarrel 1y ago
The poor guy has obviously wound himself in knots over this for several months. He went through all this, obviously really bothered by the impersonal Corporation, "solved" his problem by moving to Azure, but then wrote up this massive rant. Sounds like he needs a holiday at the least. The mere idea that reddit has a special Amazon protection team that is the cause of his account suspension is ludicrous. Hopefully this exposure will help him out.
- lazystar 1y ago
  > Hopefully this exposure will help him out. looks lile he took down his site after some other commentators in this discussion noticed some discrepencies in his resume. specifically his claims of having a PhD and a JD. Con men never learn, they just play the next fool.
  - Quarrel 1y ago
    Wow, missed this. Yikes.

reilly3000 1y ago

What a tremendous waste of resources. As a startup engineer and leader your job is to build value for customers and investors as quickly and effectively as possible, knowing the privilege of their participation with you is fleeting. The time spent in those meetings or writing this vitriolic pdf would have gone a long way towards shipping features, even if you had to survive with a suboptimal approach for a while. Time isn’t cheap and this cost a fortune.

crinkly 1y ago

Whilst this is a combination of user error and shitty support (not unexpected) I’ve seen Amazon directly not understand how their own lambda stuff works before. There is a pretty horrible bug in one of their higher level products that used lambda because they didn’t know that an execution environment can be reused between invocations. Their software assumed entirely incorrectly that it was stateless resulting in us having to, as the first line in the lambda, clean up the previous invocation’s data on the filesystem. After 3 years they haven’t fixed this as far as I am aware. Getting the actual engineers who wrote this on a call was unproductive as well suggesting it would take months to sort out. We built our own in the end.

x3n0ph3n3 1y ago
I'm not convinced AWS support didn't tell him exactly what the problem was and he ignored it. There's no reason to believe he's a reliable narrator here.
PKop 1y ago
Given that many commentors on here and reddit have already explained his mistakes, yet he has entries in his CV documenting this "bug report" and "publication" of the incident, I don't really see any evidence whatsoever that AWS didn't explain to him his mistaken understanding of how Lambda works or that they provided shitty support.

ianred 1y ago

The title should have been: AWS Lambda Works as Expected – An Application Architectural Bug [pdf] If the designer expects a reliable system that guarantees processing of the request after the return of the status, they cannot rely on a server to not crash the moment the response was sent to the wire. Serverless enforces this well, yet people try to force their view of infinitely reliable infrastructure, which is not based in reality.

PartiallyTyped 1y ago

I work at lambda, and my team would be the one to engage with OP, it most likely would have been me actually to investigate this. The fact that support didn’t engage us seems odd as we have gotten engaged for far more silly stuff. However, in this case you should have awaited for the response from event emitter. The lambda execution model clearly does not allow you to run things past the response as execution is frozen by the sandbox / virtualization software. From there, you’ve most likely caused a timeout when issuing the request.

fancyswimtime 1y ago

can aws sue this guy for product disparagement lol?

UltraSane 1y ago

The only way you can do async in Lambda is to spawn another Lambda function.

zdc1 1y ago

Taking a step back, this is an interesting reality of choosing platforms. Lambda ain't Node.js. Yes, it runs JavaScript, but that doesn't make it Node.js. JavaScript-land is full of frameworks and platforms that let you do less work in exchange for using a particular set of patterns or architecture. This also shows the best and worst of AWS: things are documented, but not always when/where/how you need them. You will still need to learn a few things the hard way when getting your toes wet (so use those "innovation tokens" sparingly).

bradyd 1y ago
Lambda's JavaScript runtime is Node.js.

WatchDog 1y ago

This is a simple misunderstanding of the Lambda execution lifecycle, but it's maybe not a great look for AWS support, that no one there was able to explain the problem to this guy. I guess it's possible someone did explain it to him, and David hasn't mentioned that in his diatribe.

jdeastwood 1y ago
I actually witnessed almost exactly the same thing happen in my org recently. AWS support was far _too_ nice and actually did escalate to the service team (or claimed to). There was a few days of back and forth which should have been resolved by saying "please read the how lambda works" section of the documentation.
- the_plus_one 1y ago
  It's been several years at this point, but I used to be a cloud support engineer for AWS (i.e. the first line of defense against support tickets). We were trained to always be kind to the customer, even if we're frustrated with them, so a response like "please read how the lambda works" probably would've gotten me reprimanded. Customers can rate your response from one to five stars, and if I remember correctly, anything less than four required a written explanation to your manager as to what went wrong. That job was a glorified call center; I didn't last very long.
ceejayoz 1y ago
> I guess it's possible someone did explain it to him, and David hasn't mentioned that in his diatribe. Not just possible. Quite likely. It seems very clear the answer was "yes, that's how it works on Lambda, and it is not going to change" and this person said "but I want it to work the other way on Lambda".

jakozaur 1y ago

It is possible to talk even to a runtime engineer, but you need to be on their enterprise plan and spend millions per year with AWS. It may sound brutal, but AWS prioritizes based on the account size. Given the AWS Active mention, the author may be living off free AWS credits and has a modest spend afterwards. Not sure if a seven-week investigation is the best use of engineering resources at early startups. Once you hit the limits of PaaS platforms, going EC2/Fargate/Kubernetes seems like a more pragmatic option. AWS Lambda is not for async jobs; once you return HTTP, it's done. Not convinced this is a bug. However, he did manage to get to the front page of Hacker News, and the write-up is quite decent (minus the AWS blame part).

icedchai 1y ago

Perhaps if he added a delay before returning the response he could’ve narrowed down the problem?

jelder 1y ago

Lambda ends when you return. That's the contract. Bro needs a dose of mechanical sympathy and intellectual vulnerability.

bfeynman 1y ago

Some devil's advocate- AWS receives many support requests that initially go to customer support engineers (in call center environments) who act as first defense for pointless simple issues. But you still have ones escalated to actual engineers and technical teams, costing hundreds or thousands of dollars. There are times when its like actual service issues and bugs etc but the number of times you have a long thread between a support engineer who does not even understand what the customer is misunderstanding (because of arrogance like this, and the customers continuing assertion they are not wrong) can go on for days/weeks and is just monumental waste of time when the answer is read the docs.

amir734jj 1y ago

Please correct me if I'm wrong but lambda model is when server responds then there shouldn't be a background task or any compute after that point otherwise how would lambda know when to stop or there wouldn't be a difference between lambda and VM.

hm-nah 1y ago

Then watch…AWS will fix it and not tell you at all. Similarly, I found that Azure Functions were saving secrets in plaintext in the SCM blade even though the Function App itself was using Key Vault References! I throughly documented the issue, reproduced it with fresh infra, filed a bug bounty, etc. Only to have Microsoft say “It’s the intended behavior” and “That’s not applicable for a bug bounty”, etc. Next month I checked the SCM area again…yeah, plaintext secrets were miraculously redacted. That’s the last time I hunt bugs for you MS!

giddily7894 1y ago

[flagged]

paulc51 1y ago
Exactly what I thought. His immediate entry in the resume about this whole circus is ludicrous: > Exposed and published a confirmed AWS Lambda runtime failure (Node.js in VPC) — silently crashing workloads post-request. Outdiagnosed L5 AWS engineers using Docker-based rebuilds, AL2023 parity, and deep forensic isolation. Case study cited on Dev.to, Hashnode, and HackerNews. He must also be completely oblivious to external feedback, given his "Case study cited on HackerNews" is all about his methodology and attitude being torn into pieces.

lijok 1y ago

You've spun yourself in knots. Next time ask chatgpt https://chatgpt.com/share/68768710-1e44-800a-9737-29772153b3d4 https://chatgpt.com/share/68768710-1e44-800a-9737-29772153b3.... I would suggest taking this down and hope it silently goes away. If you're a fractional CTO for a startup that spent a week on this, then spent god knows how long to rearchitect for Azure based on this... The startup might find this actionable. --- EDIT: After looking into this more, it seems the `context` passed to the Lambda entrypoint has a `callbackWaitsForEmptyEventLoop` parameter, which defaults to True. https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context..... Did you set this to False somewhere?

paulc51 1y ago

[dead]