10 ms·

Anti-Patterns to Avoid in Lambda Based Apps

99 points by Liriel 5y ago

kuya11 5y ago
OK, I can't let this slide. I have to call this out. 99% of this is a blatant word-for-word copy of existing AWS documentation. It's not even re-written in his own words, just straight up copied. Every bullet point is there. Every picture is a copy. Looking at Basim Hennawi's other posts on his blog, he seems to have a repeating pattern in his other posts of ripping directly from other site documentation and passing it off as his own. https://docs.aws.amazon.com/lambda/latest/operatorguide/anti-patterns.html https://docs.aws.amazon.com/lambda/latest/operatorguide/anti... https://aws.amazon.com/blogs/compute/operating-lambda-anti-patterns-in-event-driven-architectures-part-3/ https://aws.amazon.com/blogs/compute/operating-lambda-anti-p...
politelemon 5y ago
This might be a bit of #1 and bit of something else, is involving compiled languages. I see teams who normally develop in Java, will then attempt to port that entire codebase into a Lambda function and hit massive cold start times. Their solution, instead of splitting up the codebase (we need the reusability... but it's too much work to turn these parts into libraries...) is to start playing with provisioned concurrency. I think there's a mindshift required with Lambdas, which isn't immediately obvious. Anyone else encounter this and choose the easy way out - recommending they use Node/Python instead?
- chriswarbo 5y ago
  I understand your point, but I disagree with the characterisation of 'compiled versus interpreted' (which I've seen elsewhere in blog posts too). Java, C#, etc. are also interpreted languages; they just-so-happen to have a separate compile-to-bytecode step (unlike e.g. Python, which compiles to bytecode when a file is first imported); and their interpreters just-so-happen to have very slow startup times. "Compiled languages" can have very fast startup times, if we avoid languages like Java (which sacrifice a lot of startup time in favour of steady-state throughput; which is obviously inappropriate for a Lambda). If we do compile down to a native (x86_64 Linux) binary, it requires a custom runtime (e.g. if using Haskell, Rust, etc.).
  - brabel 5y ago
    > and their interpreters just-so-happen to have very slow startup times. I was surprised to find out that Common Lisp (SBCL) is so incredibly fast to start up that it can run faster than Rust even starting from source. And Lisp is the archetypical dynamic langauge! Made me question my entire career working with JVM-based languages.
    - lispm 5y ago
      Many Lisp systems like SBCL basically restore the main memory (heap) with all data, caches and native code. Then they call a start function. No need to load/link/compile/initialize code at start.
  - Zababa 5y ago
    > if we avoid languages like Java (which sacrifice a lot of startup time in favour of steady-state throughput Could you expand on that? I understand that both the JVM and Node are JIT compilers, but I don't understand why the JVM usually takes longer to start and what are the trade-offs involved here compared to Node.
    - chriswarbo 5y ago
      The main differences are probably the compilers involved, which tend to be layered based on how often a routine gets called (starting interpreted, then using a fast-but-dumb JIT compiler, then a slow-but-smart JIT compiler). There are many JVMs, each operating slightly differently, but the HotSpot JIT seems to be the most widely used (e.g. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht... ) Node runs on V8, which is described in more detail at https://v8.dev/blog/ignition-interpreter https://v8.dev/blog/ignition-interpreter Another big diffence is that Java does lots of special handling for classes, e.g. hooking up 'reflection' machinery, allowing dynamic 'class loaders', etc. which happens during an application's startup. In contrast, Javascript 'classes' are basically just functions (constructors); and Javascript functions are values just like anything else; hence there's less of an up-front cost to JS code. Python works in pretty much the same way from a language level (but its CPython interpreter works very differently to V8).
- nuerow 5y ago
  > Anyone else encounter this and choose the easy way out - recommending they use Node/Python instead? I feel your comment fails to take into account the main reason teams stick with Java in AWS Lambdas: code and dependency reuse, and consequently turnaround time. Arguing that nodejs and Golang are the ideal runtime in problems involving peeling tasks out of a monolith and into AWS Lambdas is a complete waste of time. Your goal is to offload processing tasks out of your service asap without wasting time developing and testing code written from scratch. Thus, you just create a project for your lambda, pick up your Java code that works and is well tested, offload the battle-hardened code and it's delendencies to the lambda package, add a handler to the lambda, add a lambda client to the monolith, sprinkle integ tests, and you're done: you have a production service. You're running a long-lived background task which is invoked only from time to time, and you don't really care if it takes 1 or 2 minutes to run. Additionally, do you really want to force your team to manage two distinct tech stacks just because you want to offload a process out of a service? What will that cost you?
- FreakLegion 5y ago
  Provisioned concurrency isn't worth it in my experience. You can just use, say, a CloudWatch rule to ping your desired number of instances every 5-odd minutes and keep them warm. 288 pings per day for 100 instances at 100ms per instance/ping is 2,880 seconds of execution time, or 5,760 GB-seconds at 2GB RAM (1 full vCPU and a bit). At $0.0000166667 per GB-second, that's $35 a year. You could cut that in half with smarter pings than e.g. [1], but it's not worth the engineering cycles at this price. 1. Python version: https://github.com/robhowley/lambda-warmer-py https://github.com/robhowley/lambda-warmer-py
- giorgioz 5y ago
  We do use monolithic lambdas written in Node.js and do not have cold start times problems. The problem with cold starts is more related to Java than to monolithic lambdas.
BerislavLopac 5y ago
A variant of #1 that I've encountered has a single "service" with a multiple entry-point functions, being deployed as several lambda's - one per entry point - so each lambda contains a copy of the whole codebase.
- WatchDog 5y ago
  It addresses the least privilege issue, and maybe package size (depending how intelligent your build tooling is).
- lormayna 5y ago
  You can use a Lambda Layer and avoid repeating the same code: https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-laye...
sparsely 5y ago
Neither of the suggestions for avoiding nested Lambdas seem very satisfactory - the situation described there is one which would make me consider if Lambdas were actually a good fit for my problem.
paulgb 5y ago
The “duration of invocation” payment model (discussed in #4) is ultimately what turned me off of lambda for anything that involves an external API. You could have a lambda function running smoothly, and then one day get a surprisingly high bill because a downstream service that used to respond in 100ms encountered issues and started taking 3000ms to respond so you’re suddenly paying ~30x per invocation through no fault of your own.
- eecc 5y ago
  Wouldn’t you implement that via an outbound SQS? (If that’s even possible)
- fredley 5y ago
  I wonder how long it will be before async lambdas are a thing, as in you can yield while awaiting a response - avoiding this issue.
  - red2awn 5y ago
    I think what you are describing is already here, in the form of Cloudflare Workers. See https://developers.cloudflare.com/workers/platform/limits#bundled-usage-model https://developers.cloudflare.com/workers/platform/limits#bu...
    - paulgb 5y ago
      That's pretty cool. If I were building the same service today, I'd probably use Workers.
- dragonfax 5y ago
  Any external API that could reasonably take a long time to return (because of the work its doing) should have an asynchronous API. I.e. you submit the request to start the work and can later do another request to check if its complete and get any results. Rather than waiting on a live request. Good practice for using external APIs is to NOT use any default http client settings and always provide your own timeouts to what you consider reasonable for connections and responses, as well as using a context system with deadlines so you can time out any requests that are taking more than a reasonable time to complete. Making your described surprise long expensive requests into nothing more than short errors (which hopefully you'll pick up on after a while, as long as you've got your alerting system setup right).
Zababa 5y ago
The #4 is interesting, and it makes me wonder, if the last action of a function is to call another function, will it wait for the termination? Would this be fair to call this a tail-call, and say that the lambda runtime may or may not implement TCO for its functions?
- civ0 5y ago
  It is possible to use the InvocationType "Event" [1] when invoking a Lambda. Then the API returns immediately, but you will never know if there was an error during the execution of the Lambda. The author would problaly argue to use Step Functions in this case, but using "Event" can also work fine. [1] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke https://boto3.amazonaws.com/v1/documentation/api/latest/refe...
  - andrew_ 5y ago
    Lambda + SNS + CloudWatch (with alarms) is a successful pattern for invoking other lambas very async and still being notified of problems. Swap in SQS if you're in need of message retry.
phn 5y ago
While point #1 is of course valid, it is most useful for new code, or code with significant resources behind it to optimise it for lambda. For people with an established codebase, not wanting to go all in on lambda/serverless, or wanting to use existing frameworks, the trade-off might be worth it.
giorgioz 5y ago
The whole stack of waiterio.com runs on AWS Lambda. We do use several Monolith lambdas and it works well. I believe the reason why Lambda has seen so little adoption is because cloud architects suggest to break existing services into a Lambda function for each endpoint. This causes an explosion of complexity and code that runs in the cloud in one way but it can't be reproduced locally on developer machine easily. My advice is the opposite: DO USE MONOLOTHIC LAMBDAS.
- k__ 5y ago
  I'm currently using tools like CDK and Pulumi, which allow to abstract multiple resources neatly into one component. Do you think monolitic Lambdas still make sense in that case?
  - giorgioz 5y ago
    We are using AWS CDK too! CDK is freaking AWESOME to write Infrastructure as Code. AWS CDK https://anjith-p.medium.com/writing-infrastructure-as-code-with-aws-cdk-1788ab4d1d27 https://anjith-p.medium.com/writing-infrastructure-as-code-w... Yes in my opinion the right division is GOOD 1 lambda = 1 service BAD 1 lambda = 1 endpoint I'm eager to chat with you about CDK since it's kind of new and I don't know anyone else using it. Feel free to write me at giorgio DOT zamparelli AT gmail DOT com to chat about CDK, Lambda, serverless, Infrastructure as Code.
    - jayair 5y ago
      If you are using CDK then check out SST: https://github.com/serverless-stack/serverless-stack https://github.com/serverless-stack/serverless-stack It's based on CDK and has a great local development environment for Lambda. It allows you to set breakpoints and test it locally: https://serverless-stack.com/examples/how-to-debug-lambda-functions-with-visual-studio-code.html https://serverless-stack.com/examples/how-to-debug-lambda-fu...
  - qvrjuec 5y ago
    CDK(and infra as code in general) is amazing, especially for use cases like this where one would be responsible for deploying and creating resources around 10s of lambdas for one 'service'.
WatchDog 5y ago
I don't think this is particularly good advice, he makes some valid points, but doesn't really consider the costs of the alternatives that he is suggesting. Some counterpoints: 1. Micro-lambdas How many routes does your application have 3, 30? Trying to maintain 30 different codebases and deployments would be a nightmare. Package size. Package size is unlikely to be much different by adding a little extra code, especially if the code for each route uses the same set of libraries(which will make up the bulk of the package size). Least privilege. If it makes sense to limit a routes privileges, you can simply deploy the same lambda code separately, with a different role. Upgrading. On the contrary, upgrading a lambda that uses a single codebase, is much simpler than having to deploy multiple systems and deal with the issues of interoperability and compatibility between versions. Reusing code. Also much much simpler with a monolithic codebase, the code is right there, ready to use. Testing. Testing a monolithic app is very straight-forward, you don't need to setup a complex integration test environment, all the functionality can be tested locally. 4. Lambdas calling lambas I tend to agree with this one, at least for synchronous calls, but you can also call another lambda asynchronously, actually this is something you tend to find yourself doing if you go down the path of breaking down a monolithic lambda. 5. Synchronous waiting. Yes, from a cost perspective, you should minimize the amount of time your lambda is spending waiting on external IO. However, from an application complexity perspective, you should make things as simple as possible, building overly complex event based lambda flows, can make it really difficult to debug and maintain your application.
- jackpeterfletch 5y ago
  You can solve alot of these with multi-module projects, at least in Kotlin/Java. You have a single codebase, with modules for each route, and modules for shared infra/domain code. Maven builds a JAR for each Route with only the resources that route needs, you deploy a single lambda for each JAR - but - you deploy them all together as part of the same build pipeline. Then you have, non-monolithic lambdas, that can easily be performance tuned and privileged individually, but a single codebase that represents a whole 'microservice' application in one place, with shared code, that can be upgraded and kept consistant.
- bko 5y ago
dash2 5y ago
I get the impression there is a lot of impedance mismatch between the ability to host functions in the cloud, and the tooling and ways of thinking required to make this work. Like, suddenly you can't just debug your code; debugging is magic and requires special tools. And you can't just use a recursive solution: recursion might literally cost you per call. Not that the shift is bad, but just that our tools and thinking haven't quite caught up to it. I get the same mismatch in a tiny way, when using Google colab to teach Python. It's very easy to get started, it solves lots of "local install" and versioning issues; but the keyboard shortcuts aren't quite there yet, and debugging might be scary.
- qaq 5y ago
  It is bad for most projects
- musingsole 5y ago
  I don't have the same experience. I run a web service hosted entirely on lambda with some static pages in S3. Your lambda application should be runnable locally. Doing otherwise is an implementation choice. Some errors related to underlying OS considerations are harder to catch unless you spin up a Lambda container...but, that's just it. Spin it up locally and do your debugging. Not magical at all. Recursion in an Lambda application is MORE powerful than recursion locally. With lambda, you have as much horizontal scale as you can pay for. Spin off 10,000 threads if you want. You're upset you have to pay for it? You always pay for it with hardware and electrical bills and time if not directly to AWS. My application uses a recursive lambda strategy as its core design. What's magical is that all of your logs exist cleanly in Cloudwatch and retention policies are easy.
  - __turbobrew__ 5y ago
    Being able to run the containers locally helps, but I have hit a myriad of issues which occur in the lambda runtime, but not when running the service locally. With EC2 or ECS backed by EC2 there is much less impedance mismatch between local dev environments and prod environments which results in less surprises.
stemc43 5y ago
Why use serverless to begin with? Vendor lock in, at least 5x more expensive, no good development env - what is the appeal?
- pydry 5y ago
  Not to mention all sorts of built-in limits.
- joshuanapoli 5y ago
  The expense depends on your use-case. I think that some SaaS products would be cheaper to host on Lambda. We have short bursts of high load, which led to quite a bit of waste with provisioning of traditional application servers and database. After moving from traditional servers to Lambda, we had lower hosting cost. After switching, it is easier to deploy new back-end features. It is easier to provision a sandbox for development.
  - qaq 5y ago
    so $50 a month you would save over fargate is worth all the hussle
    - joshuanapoli 5y ago
      In our case, the cost savings were significant. More important, we have operational improvements: safer delivery of new features, comprehensive metrics, and comprehensive alerting. Of course, there are many ways of achieving these goals. Kubernetes has great ways of doing the same with less vendor lock-in, but requiring more knowledge and care with the system components.
    - 0xB31B1B 5y ago
      fargate is "serverless" and has the same vendor lock in that lambda does
      - qaq 5y ago
        if you have run of the mill monolith deployed to Fargate that uses say RDS Postgres there is virtually no locking.
thecal 5y ago
Anti-pattern #1 is funny considering several people have said this is exactly how several AWS services are implemented - one Lambda function. Your deployments are much simpler. We built an application using an approach like he described and once you get to 30+ functions, it becomes a hassle.
SkeuomorphicBee 5y ago
On a slightly off-topic comment, I really wish big companies would stop cooping technical terms for their product names. I clicked thru happily expecting an article about lambda as in anonymous functions or lambda calculus in general. I know, language evolves and yada yada, but when one actor purposefully forces a new ambiguous use of a word, that is not normal language evolution, that is language vandalism. If you want to use an existing word for your product name, at least pick something from other field, so there is no ambiguity in a specific context (e.g. in a computer forum the word "mouse" unambiguously means the input device, and the word "apache" certainly means the web server or parent org).
- chestervonwinch 5y ago
  That they decided to call a DAG tool "AWS step functions" always puzzled me. I get it now -- like "step by step" -- but didn't a single person do a web search for "step function" beforehand to find it already has a totally separate meaning in mathematics and scientific applications?
  - deleted 5y ago
    [deleted]
- tokamak-teapot 5y ago
  Someone needs to have a word with Y Combinator.
- CodeAndCuffs 5y ago
  You've put one of my biggest frustrations into words very well. Someone trying to learn about serverless architecture is immediately going to run into the serverless framework which will often have nothing to do with what they're trying to accomplish. Or trying to learn about the CLI (Command Line Interface or Common Language Infrastructure) A group is trying to propose a standardized language for querying graph databases, ala SQL. They decided to call it Graph Query Language. This has nothing to do with the GQL spec, which itself has nothing to do with graph databases. The biggest obstacles in this field really feel self erected sometimes.
deleted 5y ago
[deleted]
romanhn 5y ago
This appears to be more or less a straight up copy-paste from AWS documentation (https://docs.aws.amazon.com/lambda/latest/operatorguide/anti-patterns.html https://docs.aws.amazon.com/lambda/latest/operatorguide/anti...). I'm glad to see the HN discussion because I was literally looking at those docs yesterday. Less glad about personal blogs "lifting and shifting" content and passing it off as their own.
- jareklupinski 5y ago
  perhaps they should fragment their monolithic legacy knowledge base into smaller 'micro-articles' :^)
  - romanhn 5y ago
    Funny enough, this blog post is more monolithic than the original documentation.
eranation 5y ago
The article has some good points, but just one thing about monolith lambdas. I'm sure this is not an unpopular opinion anymore (not sure if it ever was) but: 1. if you build something new, starting with a monolith lambda is OK. Your customers care about a secure, user friendly, performant product, if a monolith lambda gets it, great! 2. your code can be in a monorepo but you can still have have multiple lambdas (running the same code but each with it's own least privilege IAM policy, it's own SLA / reserved concurrency / memory settings) 3. As soon as you have something that is indeed used by more than one service / team, that it can have it's own persistence / authentication / authorization, e.g. something if there was an API/SaaS out there you would "delegate" it to it, this is when you extract a microservice. If you start with designing your microservice architecture for your side project / MVP or even seed stage startup, I applaud you, as you are succeeding in places many failed. You grow into complexity, not start with it on day one. I'm sure many agree with this sentiment but I keep seeing people getting an allergic reaction seeing any hint of mono* in any project.
adamkl 5y ago
For point number 2: > use AWS Step Functions to orchestrate these workflows using a versionable, JSON-defined state machine. I think a better approach would be to use a proper workflow engine [0]. Using a standard modelling language like BPMN and the ability to monitor, audit, and optimize make this a better option than an Amazon-specific approach. [0] https://www.infoq.com/articles/events-workflow-automation/ https://www.infoq.com/articles/events-workflow-automation/
andrew_ 5y ago
Hard disagree with the reasons given for "2. Orchestrator Lambda." While step functions are very useful and have a variety of uses, they're also VASTLY more expensive to run (the cost continues to go down but last I checked, 2.5x more expensive). Orchestrator lambdas are extremely useful, and given care in code, architecture, separation of concerns they can be extremely useful and very light. The author's point on this seems more like "I haven't seen this done elegantly" or "I tried this and my code became unmanageable," and doesn't provide solid evidence for an argument against them. The argument made could apply to any unit of infrastructure that contains code. I believe "4. Lambda functions calling Lambda functions" speaks to what I believe are the author's shortcomings for #2, with the caveat that "Cost" is definitely a consideration, as waiting for a waterfall of lambda calls will incur more cost. Call-it-and-forget-it is the more cost-efficient method for lambda-to-lambda. I trend more towards pub-sub for this scenario if the call-stack is more than 2 lambdas (e.g. lambda calling one lambda, end of stack). The author doesn't mention SNS curiously, which would suggest some inexperience in this theater. SQS can be used if payload process explicitly calls for the capabilities of SQS, but bare SNS is much more suited to light pub sub duty. Overall, this seems like a rather reductionist article written for the SEO keywords rather than content. Anyone happening across this should deep dive "the why" behind each of the claims before taking it at face value. And I wish the article stated that at the very top.

99 points by Liriel 5y ago

OK, I can't let this slide. I have to call this out. 99% of this is a blatant word-for-word copy of existing AWS documentation. It's not even re-written in his own words, just straight up copied. Every bullet point is there. Every picture is a copy. Looking at Basim Hennawi's other posts on his blog, he seems to have a repeating pattern in his other posts of ripping directly from other site documentation and passing it off as his own. https://docs.aws.amazon.com/lambda/latest/operatorguide/anti-patterns.html https://docs.aws.amazon.com/lambda/latest/operatorguide/anti... https://aws.amazon.com/blogs/compute/operating-lambda-anti-patterns-in-event-driven-architectures-part-3/ https://aws.amazon.com/blogs/compute/operating-lambda-anti-p...

politelemon 5y ago

This might be a bit of #1 and bit of something else, is involving compiled languages. I see teams who normally develop in Java, will then attempt to port that entire codebase into a Lambda function and hit massive cold start times. Their solution, instead of splitting up the codebase (we need the reusability... but it's too much work to turn these parts into libraries...) is to start playing with provisioned concurrency. I think there's a mindshift required with Lambdas, which isn't immediately obvious. Anyone else encounter this and choose the easy way out - recommending they use Node/Python instead?

chriswarbo 5y ago
I understand your point, but I disagree with the characterisation of 'compiled versus interpreted' (which I've seen elsewhere in blog posts too). Java, C#, etc. are also interpreted languages; they just-so-happen to have a separate compile-to-bytecode step (unlike e.g. Python, which compiles to bytecode when a file is first imported); and their interpreters just-so-happen to have very slow startup times. "Compiled languages" can have very fast startup times, if we avoid languages like Java (which sacrifice a lot of startup time in favour of steady-state throughput; which is obviously inappropriate for a Lambda). If we do compile down to a native (x86_64 Linux) binary, it requires a custom runtime (e.g. if using Haskell, Rust, etc.).
- brabel 5y ago
  > and their interpreters just-so-happen to have very slow startup times. I was surprised to find out that Common Lisp (SBCL) is so incredibly fast to start up that it can run faster than Rust even starting from source. And Lisp is the archetypical dynamic langauge! Made me question my entire career working with JVM-based languages.
  - lispm 5y ago
    Many Lisp systems like SBCL basically restore the main memory (heap) with all data, caches and native code. Then they call a start function. No need to load/link/compile/initialize code at start.
- Zababa 5y ago
  > if we avoid languages like Java (which sacrifice a lot of startup time in favour of steady-state throughput Could you expand on that? I understand that both the JVM and Node are JIT compilers, but I don't understand why the JVM usually takes longer to start and what are the trade-offs involved here compared to Node.
  - chriswarbo 5y ago
    The main differences are probably the compilers involved, which tend to be layered based on how often a routine gets called (starting interpreted, then using a fast-but-dumb JIT compiler, then a slow-but-smart JIT compiler). There are many JVMs, each operating slightly differently, but the HotSpot JIT seems to be the most widely used (e.g. https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.html https://cl4es.github.io/2019/11/20/OpenJDK-Startup-Update.ht... ) Node runs on V8, which is described in more detail at https://v8.dev/blog/ignition-interpreter https://v8.dev/blog/ignition-interpreter Another big diffence is that Java does lots of special handling for classes, e.g. hooking up 'reflection' machinery, allowing dynamic 'class loaders', etc. which happens during an application's startup. In contrast, Javascript 'classes' are basically just functions (constructors); and Javascript functions are values just like anything else; hence there's less of an up-front cost to JS code. Python works in pretty much the same way from a language level (but its CPython interpreter works very differently to V8).
nuerow 5y ago
> Anyone else encounter this and choose the easy way out - recommending they use Node/Python instead? I feel your comment fails to take into account the main reason teams stick with Java in AWS Lambdas: code and dependency reuse, and consequently turnaround time. Arguing that nodejs and Golang are the ideal runtime in problems involving peeling tasks out of a monolith and into AWS Lambdas is a complete waste of time. Your goal is to offload processing tasks out of your service asap without wasting time developing and testing code written from scratch. Thus, you just create a project for your lambda, pick up your Java code that works and is well tested, offload the battle-hardened code and it's delendencies to the lambda package, add a handler to the lambda, add a lambda client to the monolith, sprinkle integ tests, and you're done: you have a production service. You're running a long-lived background task which is invoked only from time to time, and you don't really care if it takes 1 or 2 minutes to run. Additionally, do you really want to force your team to manage two distinct tech stacks just because you want to offload a process out of a service? What will that cost you?
FreakLegion 5y ago
Provisioned concurrency isn't worth it in my experience. You can just use, say, a CloudWatch rule to ping your desired number of instances every 5-odd minutes and keep them warm. 288 pings per day for 100 instances at 100ms per instance/ping is 2,880 seconds of execution time, or 5,760 GB-seconds at 2GB RAM (1 full vCPU and a bit). At $0.0000166667 per GB-second, that's $35 a year. You could cut that in half with smarter pings than e.g. [1], but it's not worth the engineering cycles at this price. 1. Python version: https://github.com/robhowley/lambda-warmer-py https://github.com/robhowley/lambda-warmer-py
giorgioz 5y ago
We do use monolithic lambdas written in Node.js and do not have cold start times problems. The problem with cold starts is more related to Java than to monolithic lambdas.

BerislavLopac 5y ago

A variant of #1 that I've encountered has a single "service" with a multiple entry-point functions, being deployed as several lambda's - one per entry point - so each lambda contains a copy of the whole codebase.

WatchDog 5y ago
It addresses the least privilege issue, and maybe package size (depending how intelligent your build tooling is).
lormayna 5y ago
You can use a Lambda Layer and avoid repeating the same code: https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-laye...

sparsely 5y ago

Neither of the suggestions for avoiding nested Lambdas seem very satisfactory - the situation described there is one which would make me consider if Lambdas were actually a good fit for my problem.

paulgb 5y ago

The “duration of invocation” payment model (discussed in #4) is ultimately what turned me off of lambda for anything that involves an external API. You could have a lambda function running smoothly, and then one day get a surprisingly high bill because a downstream service that used to respond in 100ms encountered issues and started taking 3000ms to respond so you’re suddenly paying ~30x per invocation through no fault of your own.

eecc 5y ago
Wouldn’t you implement that via an outbound SQS? (If that’s even possible)
fredley 5y ago
I wonder how long it will be before async lambdas are a thing, as in you can yield while awaiting a response - avoiding this issue.
- red2awn 5y ago
  I think what you are describing is already here, in the form of Cloudflare Workers. See https://developers.cloudflare.com/workers/platform/limits#bundled-usage-model https://developers.cloudflare.com/workers/platform/limits#bu...
  - paulgb 5y ago
    That's pretty cool. If I were building the same service today, I'd probably use Workers.
dragonfax 5y ago
Any external API that could reasonably take a long time to return (because of the work its doing) should have an asynchronous API. I.e. you submit the request to start the work and can later do another request to check if its complete and get any results. Rather than waiting on a live request. Good practice for using external APIs is to NOT use any default http client settings and always provide your own timeouts to what you consider reasonable for connections and responses, as well as using a context system with deadlines so you can time out any requests that are taking more than a reasonable time to complete. Making your described surprise long expensive requests into nothing more than short errors (which hopefully you'll pick up on after a while, as long as you've got your alerting system setup right).

Zababa 5y ago

The #4 is interesting, and it makes me wonder, if the last action of a function is to call another function, will it wait for the termination? Would this be fair to call this a tail-call, and say that the lambda runtime may or may not implement TCO for its functions?

civ0 5y ago
It is possible to use the InvocationType "Event" [1] when invoking a Lambda. Then the API returns immediately, but you will never know if there was an error during the execution of the Lambda. The author would problaly argue to use Step Functions in this case, but using "Event" can also work fine. [1] https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke https://boto3.amazonaws.com/v1/documentation/api/latest/refe...
- andrew_ 5y ago
  Lambda + SNS + CloudWatch (with alarms) is a successful pattern for invoking other lambas very async and still being notified of problems. Swap in SQS if you're in need of message retry.

phn 5y ago

While point #1 is of course valid, it is most useful for new code, or code with significant resources behind it to optimise it for lambda. For people with an established codebase, not wanting to go all in on lambda/serverless, or wanting to use existing frameworks, the trade-off might be worth it.

giorgioz 5y ago

The whole stack of waiterio.com runs on AWS Lambda. We do use several Monolith lambdas and it works well. I believe the reason why Lambda has seen so little adoption is because cloud architects suggest to break existing services into a Lambda function for each endpoint. This causes an explosion of complexity and code that runs in the cloud in one way but it can't be reproduced locally on developer machine easily. My advice is the opposite: DO USE MONOLOTHIC LAMBDAS.

k__ 5y ago
I'm currently using tools like CDK and Pulumi, which allow to abstract multiple resources neatly into one component. Do you think monolitic Lambdas still make sense in that case?
- giorgioz 5y ago
  We are using AWS CDK too! CDK is freaking AWESOME to write Infrastructure as Code. AWS CDK https://anjith-p.medium.com/writing-infrastructure-as-code-with-aws-cdk-1788ab4d1d27 https://anjith-p.medium.com/writing-infrastructure-as-code-w... Yes in my opinion the right division is GOOD 1 lambda = 1 service BAD 1 lambda = 1 endpoint I'm eager to chat with you about CDK since it's kind of new and I don't know anyone else using it. Feel free to write me at giorgio DOT zamparelli AT gmail DOT com to chat about CDK, Lambda, serverless, Infrastructure as Code.
  - jayair 5y ago
    If you are using CDK then check out SST: https://github.com/serverless-stack/serverless-stack https://github.com/serverless-stack/serverless-stack It's based on CDK and has a great local development environment for Lambda. It allows you to set breakpoints and test it locally: https://serverless-stack.com/examples/how-to-debug-lambda-functions-with-visual-studio-code.html https://serverless-stack.com/examples/how-to-debug-lambda-fu...
- qvrjuec 5y ago
  CDK(and infra as code in general) is amazing, especially for use cases like this where one would be responsible for deploying and creating resources around 10s of lambdas for one 'service'.

WatchDog 5y ago

I don't think this is particularly good advice, he makes some valid points, but doesn't really consider the costs of the alternatives that he is suggesting. Some counterpoints: 1. Micro-lambdas How many routes does your application have 3, 30? Trying to maintain 30 different codebases and deployments would be a nightmare. Package size. Package size is unlikely to be much different by adding a little extra code, especially if the code for each route uses the same set of libraries(which will make up the bulk of the package size). Least privilege. If it makes sense to limit a routes privileges, you can simply deploy the same lambda code separately, with a different role. Upgrading. On the contrary, upgrading a lambda that uses a single codebase, is much simpler than having to deploy multiple systems and deal with the issues of interoperability and compatibility between versions. Reusing code. Also much much simpler with a monolithic codebase, the code is right there, ready to use. Testing. Testing a monolithic app is very straight-forward, you don't need to setup a complex integration test environment, all the functionality can be tested locally. 4. Lambdas calling lambas I tend to agree with this one, at least for synchronous calls, but you can also call another lambda asynchronously, actually this is something you tend to find yourself doing if you go down the path of breaking down a monolithic lambda. 5. Synchronous waiting. Yes, from a cost perspective, you should minimize the amount of time your lambda is spending waiting on external IO. However, from an application complexity perspective, you should make things as simple as possible, building overly complex event based lambda flows, can make it really difficult to debug and maintain your application.

jackpeterfletch 5y ago
You can solve alot of these with multi-module projects, at least in Kotlin/Java. You have a single codebase, with modules for each route, and modules for shared infra/domain code. Maven builds a JAR for each Route with only the resources that route needs, you deploy a single lambda for each JAR - but - you deploy them all together as part of the same build pipeline. Then you have, non-monolithic lambdas, that can easily be performance tuned and privileged individually, but a single codebase that represents a whole 'microservice' application in one place, with shared code, that can be upgraded and kept consistant.
bko 5y ago

dash2 5y ago

I get the impression there is a lot of impedance mismatch between the ability to host functions in the cloud, and the tooling and ways of thinking required to make this work. Like, suddenly you can't just debug your code; debugging is magic and requires special tools. And you can't just use a recursive solution: recursion might literally cost you per call. Not that the shift is bad, but just that our tools and thinking haven't quite caught up to it. I get the same mismatch in a tiny way, when using Google colab to teach Python. It's very easy to get started, it solves lots of "local install" and versioning issues; but the keyboard shortcuts aren't quite there yet, and debugging might be scary.

qaq 5y ago
It is bad for most projects
musingsole 5y ago
I don't have the same experience. I run a web service hosted entirely on lambda with some static pages in S3. Your lambda application should be runnable locally. Doing otherwise is an implementation choice. Some errors related to underlying OS considerations are harder to catch unless you spin up a Lambda container...but, that's just it. Spin it up locally and do your debugging. Not magical at all. Recursion in an Lambda application is MORE powerful than recursion locally. With lambda, you have as much horizontal scale as you can pay for. Spin off 10,000 threads if you want. You're upset you have to pay for it? You always pay for it with hardware and electrical bills and time if not directly to AWS. My application uses a recursive lambda strategy as its core design. What's magical is that all of your logs exist cleanly in Cloudwatch and retention policies are easy.
- __turbobrew__ 5y ago
  Being able to run the containers locally helps, but I have hit a myriad of issues which occur in the lambda runtime, but not when running the service locally. With EC2 or ECS backed by EC2 there is much less impedance mismatch between local dev environments and prod environments which results in less surprises.

stemc43 5y ago

Why use serverless to begin with? Vendor lock in, at least 5x more expensive, no good development env - what is the appeal?

pydry 5y ago
Not to mention all sorts of built-in limits.
joshuanapoli 5y ago
The expense depends on your use-case. I think that some SaaS products would be cheaper to host on Lambda. We have short bursts of high load, which led to quite a bit of waste with provisioning of traditional application servers and database. After moving from traditional servers to Lambda, we had lower hosting cost. After switching, it is easier to deploy new back-end features. It is easier to provision a sandbox for development.
- qaq 5y ago
  so $50 a month you would save over fargate is worth all the hussle
  - joshuanapoli 5y ago
    In our case, the cost savings were significant. More important, we have operational improvements: safer delivery of new features, comprehensive metrics, and comprehensive alerting. Of course, there are many ways of achieving these goals. Kubernetes has great ways of doing the same with less vendor lock-in, but requiring more knowledge and care with the system components.
  - 0xB31B1B 5y ago
    fargate is "serverless" and has the same vendor lock in that lambda does
    - qaq 5y ago
      if you have run of the mill monolith deployed to Fargate that uses say RDS Postgres there is virtually no locking.

thecal 5y ago

Anti-pattern #1 is funny considering several people have said this is exactly how several AWS services are implemented - one Lambda function. Your deployments are much simpler. We built an application using an approach like he described and once you get to 30+ functions, it becomes a hassle.

SkeuomorphicBee 5y ago

On a slightly off-topic comment, I really wish big companies would stop cooping technical terms for their product names. I clicked thru happily expecting an article about lambda as in anonymous functions or lambda calculus in general. I know, language evolves and yada yada, but when one actor purposefully forces a new ambiguous use of a word, that is not normal language evolution, that is language vandalism. If you want to use an existing word for your product name, at least pick something from other field, so there is no ambiguity in a specific context (e.g. in a computer forum the word "mouse" unambiguously means the input device, and the word "apache" certainly means the web server or parent org).

chestervonwinch 5y ago
That they decided to call a DAG tool "AWS step functions" always puzzled me. I get it now -- like "step by step" -- but didn't a single person do a web search for "step function" beforehand to find it already has a totally separate meaning in mathematics and scientific applications?
- deleted 5y ago
  [deleted]
tokamak-teapot 5y ago
Someone needs to have a word with Y Combinator.
CodeAndCuffs 5y ago
You've put one of my biggest frustrations into words very well. Someone trying to learn about serverless architecture is immediately going to run into the serverless framework which will often have nothing to do with what they're trying to accomplish. Or trying to learn about the CLI (Command Line Interface or Common Language Infrastructure) A group is trying to propose a standardized language for querying graph databases, ala SQL. They decided to call it Graph Query Language. This has nothing to do with the GQL spec, which itself has nothing to do with graph databases. The biggest obstacles in this field really feel self erected sometimes.

deleted 5y ago

[deleted]

romanhn 5y ago

This appears to be more or less a straight up copy-paste from AWS documentation (https://docs.aws.amazon.com/lambda/latest/operatorguide/anti-patterns.html https://docs.aws.amazon.com/lambda/latest/operatorguide/anti...). I'm glad to see the HN discussion because I was literally looking at those docs yesterday. Less glad about personal blogs "lifting and shifting" content and passing it off as their own.

jareklupinski 5y ago
perhaps they should fragment their monolithic legacy knowledge base into smaller 'micro-articles' :^)
- romanhn 5y ago
  Funny enough, this blog post is more monolithic than the original documentation.

eranation 5y ago

The article has some good points, but just one thing about monolith lambdas. I'm sure this is not an unpopular opinion anymore (not sure if it ever was) but: 1. if you build something new, starting with a monolith lambda is OK. Your customers care about a secure, user friendly, performant product, if a monolith lambda gets it, great! 2. your code can be in a monorepo but you can still have have multiple lambdas (running the same code but each with it's own least privilege IAM policy, it's own SLA / reserved concurrency / memory settings) 3. As soon as you have something that is indeed used by more than one service / team, that it can have it's own persistence / authentication / authorization, e.g. something if there was an API/SaaS out there you would "delegate" it to it, this is when you extract a microservice. If you start with designing your microservice architecture for your side project / MVP or even seed stage startup, I applaud you, as you are succeeding in places many failed. You grow into complexity, not start with it on day one. I'm sure many agree with this sentiment but I keep seeing people getting an allergic reaction seeing any hint of mono* in any project.

adamkl 5y ago

For point number 2: > use AWS Step Functions to orchestrate these workflows using a versionable, JSON-defined state machine. I think a better approach would be to use a proper workflow engine [0]. Using a standard modelling language like BPMN and the ability to monitor, audit, and optimize make this a better option than an Amazon-specific approach. [0] https://www.infoq.com/articles/events-workflow-automation/ https://www.infoq.com/articles/events-workflow-automation/

andrew_ 5y ago

Hard disagree with the reasons given for "2. Orchestrator Lambda." While step functions are very useful and have a variety of uses, they're also VASTLY more expensive to run (the cost continues to go down but last I checked, 2.5x more expensive). Orchestrator lambdas are extremely useful, and given care in code, architecture, separation of concerns they can be extremely useful and very light. The author's point on this seems more like "I haven't seen this done elegantly" or "I tried this and my code became unmanageable," and doesn't provide solid evidence for an argument against them. The argument made could apply to any unit of infrastructure that contains code. I believe "4. Lambda functions calling Lambda functions" speaks to what I believe are the author's shortcomings for #2, with the caveat that "Cost" is definitely a consideration, as waiting for a waterfall of lambda calls will incur more cost. Call-it-and-forget-it is the more cost-efficient method for lambda-to-lambda. I trend more towards pub-sub for this scenario if the call-stack is more than 2 lambdas (e.g. lambda calling one lambda, end of stack). The author doesn't mention SNS curiously, which would suggest some inexperience in this theater. SQS can be used if payload process explicitly calls for the capabilities of SQS, but bare SNS is much more suited to light pub sub duty. Overall, this seems like a rather reductionist article written for the SEO keywords rather than content. Anyone happening across this should deep dive "the why" behind each of the claims before taking it at face value. And I wish the article stated that at the very top.