24 ms·

AWS Lambda Cold Start Times

355 points by valgaze 5y ago

jmnicolas 5y ago
I just exploded in laughter when I read Java OOM at 128MB. I can't explain to my coworkers, they wouldn't understand.
- capableweb 5y ago
  Agree, it's always funny to see people using runtimes they are not used to using, so they don't know how to configure it properly, like assigning how much memory it can use. Wouldn't trust these results too much as a consequence.
mikesabbagh 5y ago
Conclusion here is to write 1 huge lambda instead of several small lambdas. right?
- davidjfelix 5y ago
  I think if you get to this point with lambda you're probably overthinking it. I think language runtime choice is important because some choices do have a cost, but likewise, choosing lambda is a tradeoff -- you don't have to manage servers, but some of the startup and runtime operations will be hidden to you. If you're okay with the possible additional latency and don't want to manage servers, it's fine. If you do and want to eke performance, it might not be. Larger lambdas mean a higher likelihood of concurrent access, which will result in cold starts when there is contention. Your cold starts will be slower with more code (It's not clear how much the size of your image affects start time, but it does have SOME impact). It's best to just not worry about these kinds of optimizations -- that's what lambda is for. If you *want* to worry about optimizing, the best optimization is running a server that is actively listening. Scope your lambda codebase in a way that makes sense. It's fine if your lambda takes multiple event types or does routing, but you're making the test surface more complex. Just like subnets, VPCs and everything else in AWS, you can scope them pretty much however you want and there's no hard fast rule saying "put more code in one" or "put less code in one", but by there are patterns that make sense and generally lots of individual transactions are easier to track and manage unless you have an explicit use case that requires scoping it to one lambda, in which case do that. There are a few cases where I've advocated for bigger lambdas vs smaller ones: * grapqhl (there still isn't a very good graphql router and data aggregator, so just handling the whole /graphql route makes the most sense) * Limited concurrency lambdas. If you have a downstream that can only handle 10 concurrent transactions but you have multiple lambda interactions that hit that service, it might be better to at least bundle all of the downstream interactions into one lambda to limit the concurrency on it.
rajin444 5y ago
Cool article, but shouldn’t Amazon be providing this kind of info? Surely they have this data internally.
- VWWHFSfQ 5y ago
  They don't want to provide it themselves because then they have to admit that the performance is abysmal. Instead they let random blogos provide this data so they can just sit back and say "you're doing it wrong."
- wongarsu 5y ago
  It makes Lambda look like a product with a much narrower niche than what AWS wants to sell it as. For many people knowing beforehand that cold start times are > 500ms with 256MB (quite extravagant for serving a single web request) would disqualify Lambda for any customer-serving endpoint. As it stands many get tricked into that choice if they don't perform these tests themselves.
  - d0gsg0w00f 5y ago
    In my experience, if you can't say "get over it" to your customers when they complain about performance then Lambda is not the right tool. Just use EC2.
    - wongarsu 5y ago
      It's an excellent product for glue code between the various AWS services. Just about every AWS product can trigger Lambda functions, so if you want to run image recognition whenever a new image is uploaded to S3, Lambda is the way to do that. They also make great cron jobs. But for some reason Amazon likes to sell it as a way to run any web application backend, as if that was a good use case.
      - sudhirj 5y ago
        It can be. We run dynamic image resizing (have a couple million image high quality originals in S3, and customers request sizes based on their screen). Each request is handled by a lambda, and even though these are memory intensive operations we never need to worry about servers or running out or RAM or circuit breakers or anything. Whatever the load it just works. The actual operations take on the order of 100ms, so the cold start is negligible to us. And the end product is cached on a CDN anyway. Costs less than one m5.large, but at peak loads it does work 100 times what's possible on the m5.large. Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance. A single servers could resize images in sequence all day and might be cheaper than running a lambda repeatedly all day, but that's not the requirement. The requirement is to suddenly do 100 things in parallel within 100ms just once when you open the app.
thdxr 5y ago
This is great work - thanks for putting this together. We recently got a request to add Rust support to ServerlessStack and looks like there's good reason to :)
muh_gradle 5y ago
Surprised to see such mediocre performance from Node. It was an engineering decision on our team to develop one of our Lambdas with Node and we were deciding between Python and Node. Looks like Go and Rust look very promising.
- Jamie9912 5y ago
  Agreed, especially when you can run Javascript without cold starts at all on Cloudflare Workers
- ignoramous 5y ago
  OP should have tested Deno (https://github.com/denoland/deno https://github.com/denoland/deno) too along side Node. Alas...
  - Zababa 5y ago
    Deno uses v8 too, that would be just comparing libuv to Tokio.
    - rough-sea 5y ago
      > just As if everything else is the same. It is a completely independent code base.
      - Zababa 5y ago
        Well, the performance seems to be mostly the same https://mayankchoubey.github.io/Deno-vs-Node-Performance/ https://mayankchoubey.github.io/Deno-vs-Node-Performance/. > It is a completely independent code base. Not exactly, the heavy lifting is done by v8 on both sides. Deno can do lots of things around the ergonomics, and switch around the event loop (though libuv is already pretty good), but outside of that they are mostly equivalent.
        deleted 5y ago
        [deleted]
davidjfelix 5y ago
You can shave even more coldstart time off the golang version by building with `go build -tags lambda.norpc` and deploying it as a custom runtime.
- pugz 5y ago
  Is that build tag necessary? I should check out what it does, because I've been deploying Go functions to custom runtimes without it. Does it skip some kind of "determine if Go-specific RPC or custom runtime" check? EDIT: I found it. Thanks for the tip! https://github.com/aws/aws-lambda-go/blob/1f782848d89c02bf802657e77fc6c7d2e50ddec2/lambda/entry.go#L64-L66 https://github.com/aws/aws-lambda-go/blob/1f782848d89c02bf80...
masklinn 5y ago
The 128M case is really strange, why do Go and Rust take so much more time to start than higher-capacity machines, and even Python? Do they get run inside a wasm runtime or something, and that runtime has to go back and forth requesting memory which a native python runtime gets “for free”?
- wongarsu 5y ago
  As far as I'm aware AWS Lambda scale various other resources with the requested memory. I assume Rust scales mostly with the assigned CPU time that increases with memory?
- zokier 5y ago
  From the looks of it, Go/Rust cold starts are almost completely CPU bound, so you see the near perfect scaling there. Meanwhile Python I'd guess is mostly io bound, which doesn't scale. That kinda makes sense as Go/Rust compile down to single executable while Python loads lots of libraries from disk. For Rust, I suspect spinning up Tokio and the rest of async runtime might be making cold starts worse. But that's purely speculation. The python lambda is good old-fashioned sync io. Another difference is that Rust lambda initializes a logger, while Python one doesn't. That might add some milliseconds too to the startup.
kidsil 5y ago
I find it surprising that NodeJS is showing the worst cold start times. Most likely over 90% of AWS Lambda functions are written NodeJS.
tyingq 5y ago
Nice to have some updated data and comparisons. This article doesn't include the effect if the Lambda has to connect to a VPC though, which adds time for the ENI. Though that was greatly improved in 2019-2020: https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/ https://aws.amazon.com/blogs/compute/announcing-improved-vpc...
_fat_santa 5y ago
At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint for the first time, it would take 2x as long as it normally would at first. To combat this we wrote a "runWarm" function that kept the API alive at all times. Sure kind of defeats the purpose of serverless but hey, enterprise software.
- devonkim 5y ago
  To be fair, all you’d need to accomplish that without more than necessary parts for production is to ensure that the code path invoking the function is accessed via an external monitoring probe with an adjustment to SLA or SLO to account for the cold start time. Obviously not going to work for many systems, but it’s easy to forget all the side effects of the observability plane when writing applications.
- deleted 5y ago
  [deleted]
- nodejs_rulez_1 5y ago
  > ...Sure kind of defeats the purpose of serverless but hey, enterprise software. No, no, no. It's "hey, Amazon/Microsoft cloud engineering". They should be amazing with whiteboard interview exercises though.
- jpdb 5y ago
  > To combat this we wrote a "runWarm" function that kept the API alive at all times. This doesn't really work like you'd expect and isn't recommended, as it only helps a particular use-case. The reason is that AWS Lambda will only keep a single instance of your function alive. That means if two requests come in at the same time, you'd see a cold start on one of those invocations. Instead, you want to look at something like provisioned concurrency.
  - VWWHFSfQ 5y ago
    Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely better. I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.
jabart 5y ago
We run a few .net core lambdas and a few things that make a big difference for latency. 1. pre-jit the package, this reduces cold start times as the JIT doesn't need to run on most items. Still does later to optimize some items. 2 is sticking to the new .net json seralizer. The reference code uses both the new and old newtsonsoft package. The old package has higher memory allocations as it doesn't make use of the Span type.
- moonchrome 5y ago
  Great tip for running the JIT AOT, for anyone interested Microsoft calls this "ReadyToRun" compilation [1]. [1] https://docs.microsoft.com/en-us/dotnet/core/deploying/ready-to-run https://docs.microsoft.com/en-us/dotnet/core/deploying/ready... I wonder did you test if the increased size results in an actual win for the startup time ?
  - jabart 5y ago
    After like 256mb it is less of an impact when using readytorun. Some of the lambdas I have are webhooks so latency isn't as important and when it's user facing 512mb seems to be a sweet spot.
  - Shadonototra 5y ago
    and then you need to upgrade the plan because it is disk/memory hungry and doesn't fit within the 128mb plan, so you'll pay 4x more already just use GO and be faster while saving money at the same time
    - moonchrome 5y ago
      Like someone else said lambdas are priced by memory * execution time - if you cut execution time by 1/2 by doubling the memory and using AOT - you got faster lambdas for free (even if it's not as high as 1/2 you'll probably still not be paying x2).
  - gokhan 5y ago
    The test code is quite small and might not benefit from R2R that much, libs it relies on are already jitted. Ditching Newtonsoft would affect response time though.
unraveller 5y ago
just the cold-start data: https://gist.github.com/Aleksandr-Filichkin/925fce9d910e04d2037f18caf3cfc3a5/raw/44d39a5091c822aa1d653246413cf06b404adc01/Cold-start..tsv https://gist.github.com/Aleksandr-Filichkin/925fce9d910e04d2... mirror: https://scribe.rip/@filia-aleks/aws-lambda-battle-2021-performance-comparison-for-all-languages-c1b441005fd1 https://scribe.rip/@filia-aleks/aws-lambda-battle-2021-perfo...
moonchrome 5y ago
One thing I wish he included was .NET 5 - since that switched to using Docker images I would be very interested in the differences.
- ewjt 5y ago
  Lambda only has default support for the LTS releases of .NET, which is 2.1, 3.1 and the upcoming (November 2021) .NET 6. The only way to run .NET 5 in a Lambda that I know of would be custom runtimes or containers. Is that what do you mean by “switched to using Docker images”? This article describes the containerized performance of .NET in Lambda and the cold starts are dramatically (~4x) worse. https://www.kloia.com/blog/aws-lambda-container-image-.net-benchmark https://www.kloia.com/blog/aws-lambda-container-image-.net-b...
  - moonchrome 5y ago
    >Lambda only has default support for the LTS releases of .NET Not true, https://aws.amazon.com/blogs/developer/net-5-aws-lambda-support-with-container-images/ https://aws.amazon.com/blogs/developer/net-5-aws-lambda-supp... >Even though Lambda’s policy has always been to support LTS versions of language runtimes for managed runtimes, the new container image support makes .NET 5 a first class platform for Lambda functions.
sp33der89 5y ago
I would love to see languages like OCaml, D, Nim benchmarked here as well. They sit sortof in between Go and Rust, where I don't have to deal with manual memory management but get enough expressiveness to write a nice Lambda.
- ignoramous 5y ago
  Those, and: Zig would provide for an interesting contrast with Rust and Nim.
- throwaway894345 5y ago
  Not sure about the others, but OCaml tends to perform in the same ballpark as Go, and a bit slower for programs that benefit from shared memory parallelism.
cloudytechi147 5y ago
Many organizations use Linux OS to run file servers, print servers, content delivery systems, global caching servers, data archives, VPN servers, etc. Surely, Windows and macOS are easy to use, but Linux distributions have become more classy and user-friendly over the years. Linux is also considered to be more secure than Windows and macOS and makes software deployment quite easy. There are certain Linux distros that support enterprise-related tasks, and for that, you can consider these options: Blog link: https://webhostingprime.com/best-linux-os/ https://webhostingprime.com/best-linux-os/
psanford 5y ago
Something I discovered recently, for my tiny Go Lambda functions it is basically always worth it to run them at least with 256mb of memory even if they don't need more than 128mb. This is because most of my functions run twice as fast at 256mb than they do at 128mb. Since lambda pricing is memory_limit times execution time, you get better performance for free. Test your lambda functions in different configurations to see if the optimal setting is different than the minimal setting.
- jSherz 5y ago
  CPU allocated to Lambda functions scales with the memory allowance [1] and you can use power tuning [2] to find the optimal values. [1] https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html https://docs.aws.amazon.com/lambda/latest/dg/configuration-f... [2] https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:451282441545:applications~aws-lambda-power-tuning https://serverlessrepo.aws.amazon.com/applications/arn:aws:s...
xwdv 5y ago
If cold start times are an issue use Cloudflare workers instead, they’re always warm.
- Jonovono 5y ago
  oh baby, they are warm indeed
cmcconomy 5y ago
My experience with cold starts in Azure Functions Serverless is pretty awful. Like most other Azure services, their affordable consumer grade offerings are designed from the ground up not to be good enough for "serious" use. Cold start times compared to Lambda are worse, and in addition, we would get random 404s which do not appear in any logs; inspecting these 404s indicated they were emitted by nginx, leading me to believe that the ultimate container endpoint was killed for whatever reason but that fact didn't make it back to the router, which attempted and failed to reach the function. Of course the cold start and 404 are mitigated if you pay for the premium serverless or just host their middleware on their own App Service plans (basically VMs)
- secondaryacct 5y ago
  My company is moving away from datacenter and into Azure and I have to get the az900 this week, it doesnt bode well. And I was so happy to leave the clusterfuck of 300 aws lambda I was working with in my prev company. What an expensive fad, and no engineer is ever consulted ...
  - cmcconomy 5y ago
    There are things I like, such as the consistent and pervasive security model with Azure AD. I don't even mind running stuff on App Services. I just find their commodity serverless offering in particular is subpar
    - meekins 5y ago
      > consistent and pervasive security model with Azure AD Wait, this is the first time I hear this about Azure. Could you elaborate? It is possible that things have improved significantly since I last worked with Azure but lack of a consistent security model (like IAM on AWS) to control human and service (Azure Functions, App Service apps etc) access to specific resources (Cosmos databases, EventHubs etc) especially painful.
      - cmcconomy 5y ago
flurie 5y ago
I would have liked to see more values along the lambda "breakpoints" between 1GB and 10GB of memory. Unless things have changed recently, my understanding is that CPU and IO scale up specifically at those breakpoints rather than being continuous.
losvedir 5y ago
I'm surprised Node has cold-start issues. I had it in my mind that JS was Lambda's "native" language and wouldn't have cold start issues at all. Did it used to be like that? Didn't Lambda launch with only support for JS, and maybe a couple other languages that could compile to it?
- yoava 5y ago
  With node.js, the cold start problem is caused by how node loads files. For each file it does about 10 IO operations (to resolve the file from the module name), then load, parse and compile the file. If using any file system that is not super fast, this amounts to long delays. There are ways to get around that, but those are not available on lambda
- maxmcd 5y ago
  I thought nodejs/v8 or any javascript runtime would have some kind of startup cost since it has to parse and compile the javascript code first. See a simple hello world execution time comparison: # a Go hello world $ time ./hello hi real 0m0.002s $ time echo 'console.log("hello")' | node - hello real 0m0.039s The ~25ms of cold start noted in this article feels acceptable and impressive to me, given what node is doing under the hood.
- artimaeis 5y ago
  Yeah, at launch Lambda only supported Node.js. https://web.archive.org/web/20141115183837/http://aws.amazon.com/lambda/faqs/ https://web.archive.org/web/20141115183837/http://aws.amazon...
- seniorsassycat 5y ago
  I wonder ho w much time was spent requiring all of aws-sdk. The v3 sdk is modular and should be quicker to load. Bundlers like rebuild save space and reduce parsing time.
  - lukeramsden 5y ago
    Double digit ms in my experience. Not including the fact that V2 will renegotiate TLS constantly when it could just keep the socket open.
_wldu 5y ago
It's really amazing to see how Lambda has grown to support all of these languages.
haolez 5y ago
Slightly off topic, but what's the deal with Azure Functions cold start times in the Consumption (i.e. serverless) plan? I get cold start times in the multi seconds range (sometimes huge values, like 20s). Am I doing something wrong? Or is this expected?
- kkielhofner 5y ago
  I've experienced this as well. I gave up on optimizing it. I get around it by using a load balancer (Cloudflare currently) that does periodic health checks. Keeps it alive and the charges are minimal (still well within the free tier). Speaking of Cloudflare it's on my "nice to-do" list to move this to Workers as a primary anyway. I also use a completely separate uptime monitor and alerting platform (Uptime Robot) so one way or another I'd be keeping at least one instance warm no matter what.
Shadonototra 5y ago
that is why the language for cloud native is GO, and languages like Java/C# are dead tech stack, they failed to reinvent themselves to stay relevant
- FpUser 5y ago
  >"they failed to reinvent themselves to stay relevant" This sounds like it came straight out of "The Corporate BS generator" - https://www.atrixnet.com/bs-generator.html https://www.atrixnet.com/bs-generator.html.
  - Shadonototra 5y ago
    no, it's the direct translation of the benchmark of this post cold start should be a thing of the past, hence the "failed to reinvent themselves to stay relevant" you need to speak to the hardware directly, not to the VM or the embedded compiler with ARM and ultimately RISC-V coming, that's gonna be even more true
    - ewjt 5y ago
      How does a single metric from a highly specialized runtime environment indicate a tech stack is dead? There are things you can do right now [1] to mitigate these cold start issues. Going forward, ahead-of-time compilation will be an option. [2] Aside from cold starts, note that the improvements in .NET make ASP.NET Core one of the fastest web frameworks. [3] The article: > “.Net has almost the same performance as Golang and Rust, but only after 1k iterations(after JIT).” Additions like async/await and nullable reference types make it easier to write bug-free code, which for a lot of folks is a better trade off than “speaking to the hardware directly”. .NET also runs natively on a bunch of platforms now, including ARM. I’d call all of that continuous improvement. Perhaps even reinvention? [1] https://docs.microsoft.com/en-us/dotnet/core/deploying/ready-to-run https://docs.microsoft.com/en-us/dotnet/core/deploying/ready... [2] https://github.com/dotnet/runtimelab/tree/feature/NativeAOT https://github.com/dotnet/runtimelab/tree/feature/NativeAOT [3] https://www.techempower.com/benchmarks/#section=test&runid=57b25c85-082a-4013-b572-b0939006eaff&hw=ph&test=composite&a=2 https://www.techempower.com/benchmarks/#section=test&runid=5...
projectileboy 5y ago
AWS Lambda is pretty cool, it just gets used a lot for applications that it was never really designed for. While I wish that Amazon would address the cold start times, if you try to grill your burgers with a cordless drill, you can’t really blame the drill manufacturer when the meat doesn’t cook.
rcarmo 5y ago
I recently discovered that uWSGI has a "cheap mode" that will hold the socket open but only actually spawn workers when a connection comes in (and kill them automatically after a timeout without any requests). Pertinent options: https://github.com/piku/piku/blob/master/piku.py#L908 https://github.com/piku/piku/blob/master/piku.py#L908 If you already have 24/7 compute instances going and can spare the CPU/RAM headroom, you can co-host your "lambdas" there, and make them even cheaper :)
tinyprojects 5y ago
If anyone is running into cold start problems on Firebase, I recently discovered you can add .runWith({minInstances: 1}) to your cloud functions. It keeps 1 instance running at all times, and for the most part completely gets rid of cold starts. You have to pay a small cost each month (a few dollars), but its worth it on valuable functions that result in conversions, e.g. loading a Stripe checkout.
- digianarchist 5y ago
  This is basically how Google Cloud Run works if I'm not mistaken.
  - fulafel 5y ago
    And AWS Lambda "provisioned concurrency".
  - prattmic 5y ago
    Google Cloud Run also scales to zero by default, but you can configure the minimum number of instances: https://cloud.google.com/run/docs/configuring/min-instances https://cloud.google.com/run/docs/configuring/min-instances
fulafel 5y ago
Container based lambda image configurations (vs zip based) would be a good addition to this comparison. People use them eg to get over the zip based lambda size limit. Also maybe mentione provisioned concurrency (where you pay AWS to keep one or more instances of your lambda warm). Both of these are supported by Serverless framework btw.
- time0ut 5y ago
  Definitely. In my experience, docker image based Lambdas had consistently poor (>3s) cold starts regardless of memory. I hope it will eventually improve as it is a much nicer packaging approach than ZIP file. Also, it would have been nice to include ARM vs x86 now that ARM is available.
e3bc54b2 5y ago
I just want to appreciate the article. Starting with non-clickbait title, upfront summary, detailed numbers, code for reruns, great graphs, no dreamy story and no advertisement of any kind. It is hosted on Medium but the author has done a banging great job, so gets a pass. If he is reading, excellent work!
gunnarmorling 5y ago
The best cold starts are those which aren't noticed by the user. For my blog search (which runs on Lambda), I found a nice way of achieving that [1]: as soon as a user puts the focus to the input field for the search text, this will already submit a "ping" request to Lambda. Then, when they submit the actual query itself, they will hit the already running Lambda most of the times. And, as others said, assigning more RAM to your Lambda than it actually may need itself, will also help with cold start times, as this increases the assigned CPU shares, too. [1] https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-blog/ https://www.morling.dev/blog/how-i-built-a-serverless-search...
- buzzdenver 5y ago
  So effectively you're going to pay double?
  - musingsole 5y ago
    You increase the usage of the lambda, sure, but not by double unless all this lambda does is respond to the search. There's a point where this is all semantics when running cloud services with regard to how you've deployed your code behind gateways and lambdas, but most applications have obvious triggers that could warm up a lambda at 10% or less overhead. And even then....lambdas are dirt cheap.
    - wpietri 5y ago
      Lambdas can be dirt cheap if you use them well. But that's always what happens. I recently saw a report about contractor who used them for crawling websites. Their client saw a surprise $12k bill for Lambda, when a simple Scrapy crawler on a low-end instance would have cost them < $100/month for the same load. Why? Because if you spin up a ton of Lambda invocations and have them sit around waiting for the network, you pay for each CPU to sit idle, rather than having just one CPU stay busy managing a bunch of async IO.
      - laurencerowe 5y ago
davewritescode 5y ago
The main downside of Lambda, in particular for user facing applications is that the incentives of the cloud provider and you are completely opposed. You (the developer) want a bunch of warm lambdas ready to serve user requests and the cloud provider is looking to minimize costs by keeping the number of running lambdas as low as possible. It's the incentive model that fundamentally makes Lambda a poor choice for these types of applications. Other downsides include the fact that Lambdas have fixed memory sizes. If you have units of work that vary in amount of memory required you're basically stuck paying the costs of the largest units of work unless you can implement some sort of routing logic somewhere else. My company ran into this issue using lambdas to process some data where the 99% of requests were fine running in 256mb but a few required more. There was so way to know ahead of time how much memory the computation would require ahead of time. We ended up finding a way to deal with it but in the short term we had to bump the lambda memory limits. That doesn't even get into the problems with testing. In my experience, Lambdas are best used as glue between AWS components, message processors and cron style tasks.
- paxys 5y ago
  Disagree with "completely opposed". Cloud providers want to make money, sure, but in general everyone in the ecosystem benefits if every CPU cycle is used efficiently. Any overhead goes out of both AWS's and your pockets and instead to the electricity provider, server manufacturer, cooling service.
- joncrane 5y ago
  > the incentives of the cloud provider and you are completely opposed I think this is a little overstated. The cloud provider wants their customers to be happy while minimizing costs (and therefore costs to the customer). It's not truly a perverse incentive scenario.
- tuananh 5y ago
  > In my experience, Lambdas are best used as glue between AWS components, message processors and cron style tasks. i think lambda is the defacto standard now for integration on aws. but they are pushing lambda hard for other workload too
l8again 5y ago
I wish the author had done a comparison for Java apps using JLink that generates a custom Java runtime image that contains only the platform modules that are required for a given application, and if that makes a difference.
- gunnarmorling 5y ago
  jlink usage won't make any significant difference typically. What does make a difference is (App)CDS though, as available in newer Java versions. Memory-mapping a file with the class metadata from previous runs can easily shave off a second or more from time-to-first-response [1], depending on number and size of classes required for that. [1] https://www.morling.dev/blog/smaller-faster-starting-container-images-with-jlink-and-appcds/ https://www.morling.dev/blog/smaller-faster-starting-contain...
Kalanos 5y ago
i read that they offer C as well. wonder what those times look like
- eldavido 5y ago
  It's low (100-200ms). Read my post above
eldavido 5y ago
C++. 200ms cold start. provided.al2 runtime environment. Lambda success story: Started with a .NET Core API about a year ago. Monolith-first. Mix of clients across mobile and React. Async/await is one of the better things about C# (the language used for ASP.NET Core) and as a result, we were able to do things you'd never consider doing in-process on a system like Ruby on Rails (right on the thread serving the HTTP request), like transcoding a 12 megapixel HEIC upload into JPEG. We just did it, left the connection open, and when it was done, returned an HTTP 200 OK. That worked well for a while and let us serve tons of clients on a single Heroku dyno. The problem: memory. Resizing images takes tens/hundreds of MB when you're doing it into three different formats. Over the last two weeks, I extracted the HEIC->JPEG transcode/resize out of our monolith into a Lambda. I'm extremely happy with how it turned out. We went with C++ because the whole idea was performance, we're going to be doing point cloud processing and other heavyweight stuff, and wanted fine-grained control of memory. Our process has 28MB of dynamic libraries (.so files), starts in 200ms, and runs comfortably on a 512MB instance. We moved to 1024 to provide a margin of safety just in case we get a really large image. The system has progressed into "I don't even think about it"-level reliably. It just works and I pay something like $1 for 40-50k transcode operations. No EC2 instances to manage, no queues, no task runners, no Ruby OOM, no running RabbitMQ, none of that (former ops engineer at a very high-scale analytics company). As a general comment, I don't see many cloud services written in C/C++. This is no doubt partly because those skills just aren't widespread. But I think the bigger lesson is that it might be worth adding a little bit of development complexity to save 10x as much ops overhead. When I explained this setup to my friend, his first reaction was, "Why didn't you just put ImageMagick (the binary) into a container?" Once I explained that actually, I need to get images from S3, and write them into several formats, and manipulate their S3 keys in somewhat complex ways, fire off an HTTP request to a server, and pass a JWT around...sure, I could write this in a shell script, with wget, and curl, and everthing else. But at some point you just have to write the right code for the job using the right tools. I think hybrid approaches like this make the most sense. .NET and Java are great high-productivity tools for running server apps where memory is relatively abundant. I wouldn't try to move a system like that onto Lambda any more than I'd try to do something that more naturally fits with a queue/worker pattern on a webserver. This seems kind of obvious but if I'm being honest, it's probably experience talking a bit. It's also neat to just get back to the metal a bit. Drop the containers, runtime environments, multi-hundred-MB deployment packages, just ship a xx MB package up to the cloud, deploy it, and have it run as a standalone linux binary with all the speed and simplicity that brings. Modern C++ is a totally different animal than 90s C++, I'd encourage giving it a try if you haven't in a while.
NicoJuicy 5y ago
Any experience with Cloudflare workers? They are supposed to have 0 cold start time: https://blog.cloudflare.com/eliminating-cold-starts-with-cloudflare-workers/ https://blog.cloudflare.com/eliminating-cold-starts-with-clo...
erikerikson 5y ago
I was surprised by the quality of this one. That said... Cold starts are a FaaS learning subject but they almost never matter much in practice. What workloads are intermittent and also need extremely low latencies? Usually when I see people worrying about this it is because they have architected their system with call chains and the use case, if it really matters, can be re-architected so that the query result is pre prepared. This is much like search results... Search engines certainly don't process the entire web to service your queries. Instead, they pre-calculate the result for each query and update those results as content CRUD happens.
somethingAlex 5y ago
Honestly shocked that rust is about 4 times faster than node for the DynamoDB insert workload in the average case. I knew it'd be faster but I would have expected maybe 50% faster since most of the time is probably spent simply sending the data to dynamoDB and awaiting a response. Also, what is up with Python being faster than Node in the beginning and then getting slower over time? The other languages (apart from graal) get faster over time. I'm referring to the average MS latency at 128MB graph at the bottom.
- deleted 5y ago
  [deleted]
Havoc 5y ago
Where feasible I’m using cloudflare workers (and KV) instead to be honest. Less versatile but no tangible cold start time
yoava 5y ago
The article states that 600ms is a low cold start figure. However, 600ms cold start is still unacceptable for usage by WebApps. For webapp, the figure should be 100ms. The only platform that meets that figure (that I know of) is Velo by Wix with around 50ms cold start for node.js
bezossucks 5y ago
If you can run your entire function in V8 and don't need node, Cloudflare Workers is MUCH faster, more affordable, more manageable, and more reliable. They get cloned all over the world and you're not region-locked. https://www.cloudflare.com/learning/serverless/serverless-performance/ https://www.cloudflare.com/learning/serverless/serverless-pe...
- wertgklrgh 5y ago
  yet nobody ever uses them
- StanAngeloff 5y ago
  We've moved from AWS to Cloudflare and I can confirm Workers are awesome. The only downside is lots of work required to get npm packages running properly. Due to the lack of Node.js' runtime, bundling, shims and stubbing required to get stuff like "fs" and "net" compiling. AWS is more mature, has a better community support around it and generally the tooling is more stable. Workers are quickly catching up, though, with great new additions such as ES modules, custom builds, Miniflare, etc.
rimutaka 5y ago
The articles states that it takes approximately 1000 requests to optimize / warm up. I suspect that they had concurrency set at the default 999, so the first 999 requests would spin up new instances. Does that mean their 15,000 requests were actually 15 requests spread over 1000 instances?
The_rationalist 5y ago
It's unfortunate than ZGC and Shenandoah weren't tested nor parallel gc
asdev 5y ago
This guy has such a rudimentary understanding that he can't point to real examples of Ethereum applications that support his point. Most smart contracts are immutable and non-upgradeable, which nullifies this entire blog post. Many protocols are also moving to DAO governance for even further decentralization of power. This reflects a highly pervasive issue as people with a somewhat technical background but no experience with crypto networks and applications think they are qualified to give opinions on whether the protocols will make it or not.
mulmboy 5y ago
A pattern I have implemented is to have my API code on both ECS/Fargate and Lambda at the same time, and send traffic to the appropriate one using an Elastic Load Balancer. I flag specific endpoints as "cpu intensive" and have them run on lambda. Implemented by - Duplicating all routes in the API with the "/sls/" prefix (this is a couple of lines in FastAPI) - Setting up a rule in ELB to route to Lambda if the route starts with /sls, or to ECS otherwise. - Set up the CPU intensive routes to automatically respond with a 307 to the same route but prefixed with /sls. Boom, with that the system can handle bursts of CPU intensive traffic (e.g. data exports) while remaining responsive to the simple 99% of requests all on one vCPU. And the same dockerfile, with just a tiny change, can be used both in ECS and Lambda.
tiew9Vii 5y ago
The Rust metrics are interesting. I've been getting around 20ms cold starts, 1ms warm exec on the 128MB ARM Graviton2 using Rust for the most basic test cases. Graviton2 was slightly slower on cold starts than X86 for me (1-2ms) but who doesn't want to save $0.0000000004 per execution? Adding calls to parameter store/dynamo DB bumps it up a little but still < 120ms cold, and any added latency comes from waiting on the external service calls. Memory usage is 20-30MB, and I haven't done anything to optimise memory. I know I can get rid of a few allocations I'm doing for simplicity if I want to. I've not always been the greatest fan of Lambdas seeing it has hidden complexity orchestrating and a blackbox for debugging. Re-visiting a few years on and with Rust, you get an excellent language, excellent runtime characteristics and substantial cost savings unless you really need more than 128MB memory, i.e. processing large volumes of data per execution in memory or transcoding. Any asynchronous/event-driven service I write, I'll just package as a Rust lambda going forward and pay fractions of a cent per month. I am still on-the-wall with HTTP exposed services as that's a big plumbing exercise and hidden gateway costs but not as adverse to it as I was.
bilalq 5y ago
> NodeJs is the slowest runtime, after some time it becomes better(JIT?) but still is not good enough. In addition, we see the NodeJS has the worst maximum duration. The conclusion drawn about NodeJS performance is flawed due to a quirk of the default settings in the AWS SDK for JS compared to other languages. By default, it opens and closes a TCP connection for each request. That overhead can be greater than the time actually needed to interact with DDB. I submitted a pull request to fix that configuration[0]. I expect the performance of NodeJS warm starts to look quite a bit better after that. [0]: https://github.com/Aleksandr-Filichkin/aws-lambda-runtimes-performance/pull/6 https://github.com/Aleksandr-Filichkin/aws-lambda-runtimes-p...
- bilalq 5y ago
  In addition, the NodeJS cold start time can be further optimized by bundling into a single file artifact to reduce the amount of disk IO needed when requiring dependencies. Webpack, Parcel, ESBuild, and other bundlers could achieve that, I'm sure. EDIT: That may already be happening here in the build.sh file. I see it runs `sam build --use-container NodeJsFunction -b nodejs`.
paulmendoza 5y ago
I wonder if he used the special .net compilation setting that reduces quick starts. It requires compiling the function on Amazon Linux 2

355 points by valgaze 5y ago

jmnicolas 5y ago

I just exploded in laughter when I read Java OOM at 128MB. I can't explain to my coworkers, they wouldn't understand.

capableweb 5y ago
Agree, it's always funny to see people using runtimes they are not used to using, so they don't know how to configure it properly, like assigning how much memory it can use. Wouldn't trust these results too much as a consequence.

mikesabbagh 5y ago

Conclusion here is to write 1 huge lambda instead of several small lambdas. right?

davidjfelix 5y ago
I think if you get to this point with lambda you're probably overthinking it. I think language runtime choice is important because some choices do have a cost, but likewise, choosing lambda is a tradeoff -- you don't have to manage servers, but some of the startup and runtime operations will be hidden to you. If you're okay with the possible additional latency and don't want to manage servers, it's fine. If you do and want to eke performance, it might not be. Larger lambdas mean a higher likelihood of concurrent access, which will result in cold starts when there is contention. Your cold starts will be slower with more code (It's not clear how much the size of your image affects start time, but it does have SOME impact). It's best to just not worry about these kinds of optimizations -- that's what lambda is for. If you *want* to worry about optimizing, the best optimization is running a server that is actively listening. Scope your lambda codebase in a way that makes sense. It's fine if your lambda takes multiple event types or does routing, but you're making the test surface more complex. Just like subnets, VPCs and everything else in AWS, you can scope them pretty much however you want and there's no hard fast rule saying "put more code in one" or "put less code in one", but by there are patterns that make sense and generally lots of individual transactions are easier to track and manage unless you have an explicit use case that requires scoping it to one lambda, in which case do that. There are a few cases where I've advocated for bigger lambdas vs smaller ones: * grapqhl (there still isn't a very good graphql router and data aggregator, so just handling the whole /graphql route makes the most sense) * Limited concurrency lambdas. If you have a downstream that can only handle 10 concurrent transactions but you have multiple lambda interactions that hit that service, it might be better to at least bundle all of the downstream interactions into one lambda to limit the concurrency on it.

rajin444 5y ago

Cool article, but shouldn’t Amazon be providing this kind of info? Surely they have this data internally.

VWWHFSfQ 5y ago
They don't want to provide it themselves because then they have to admit that the performance is abysmal. Instead they let random blogos provide this data so they can just sit back and say "you're doing it wrong."
wongarsu 5y ago
It makes Lambda look like a product with a much narrower niche than what AWS wants to sell it as. For many people knowing beforehand that cold start times are > 500ms with 256MB (quite extravagant for serving a single web request) would disqualify Lambda for any customer-serving endpoint. As it stands many get tricked into that choice if they don't perform these tests themselves.
- d0gsg0w00f 5y ago
  In my experience, if you can't say "get over it" to your customers when they complain about performance then Lambda is not the right tool. Just use EC2.
  - wongarsu 5y ago
    It's an excellent product for glue code between the various AWS services. Just about every AWS product can trigger Lambda functions, so if you want to run image recognition whenever a new image is uploaded to S3, Lambda is the way to do that. They also make great cron jobs. But for some reason Amazon likes to sell it as a way to run any web application backend, as if that was a good use case.
    - sudhirj 5y ago
      It can be. We run dynamic image resizing (have a couple million image high quality originals in S3, and customers request sizes based on their screen). Each request is handled by a lambda, and even though these are memory intensive operations we never need to worry about servers or running out or RAM or circuit breakers or anything. Whatever the load it just works. The actual operations take on the order of 100ms, so the cold start is negligible to us. And the end product is cached on a CDN anyway. Costs less than one m5.large, but at peak loads it does work 100 times what's possible on the m5.large. Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance. A single servers could resize images in sequence all day and might be cheaper than running a lambda repeatedly all day, but that's not the requirement. The requirement is to suddenly do 100 things in parallel within 100ms just once when you open the app.

thdxr 5y ago

This is great work - thanks for putting this together. We recently got a request to add Rust support to ServerlessStack and looks like there's good reason to :)

muh_gradle 5y ago

Surprised to see such mediocre performance from Node. It was an engineering decision on our team to develop one of our Lambdas with Node and we were deciding between Python and Node. Looks like Go and Rust look very promising.

Jamie9912 5y ago
Agreed, especially when you can run Javascript without cold starts at all on Cloudflare Workers
ignoramous 5y ago
OP should have tested Deno (https://github.com/denoland/deno https://github.com/denoland/deno) too along side Node. Alas...
- Zababa 5y ago
  Deno uses v8 too, that would be just comparing libuv to Tokio.
  - rough-sea 5y ago
    > just As if everything else is the same. It is a completely independent code base.
    - Zababa 5y ago
      Well, the performance seems to be mostly the same https://mayankchoubey.github.io/Deno-vs-Node-Performance/ https://mayankchoubey.github.io/Deno-vs-Node-Performance/. > It is a completely independent code base. Not exactly, the heavy lifting is done by v8 on both sides. Deno can do lots of things around the ergonomics, and switch around the event loop (though libuv is already pretty good), but outside of that they are mostly equivalent.
      - deleted 5y ago
        [deleted]

davidjfelix 5y ago

You can shave even more coldstart time off the golang version by building with `go build -tags lambda.norpc` and deploying it as a custom runtime.

pugz 5y ago
Is that build tag necessary? I should check out what it does, because I've been deploying Go functions to custom runtimes without it. Does it skip some kind of "determine if Go-specific RPC or custom runtime" check? EDIT: I found it. Thanks for the tip! https://github.com/aws/aws-lambda-go/blob/1f782848d89c02bf802657e77fc6c7d2e50ddec2/lambda/entry.go#L64-L66 https://github.com/aws/aws-lambda-go/blob/1f782848d89c02bf80...

masklinn 5y ago

The 128M case is really strange, why do Go and Rust take so much more time to start than higher-capacity machines, and even Python? Do they get run inside a wasm runtime or something, and that runtime has to go back and forth requesting memory which a native python runtime gets “for free”?

wongarsu 5y ago
As far as I'm aware AWS Lambda scale various other resources with the requested memory. I assume Rust scales mostly with the assigned CPU time that increases with memory?
zokier 5y ago
From the looks of it, Go/Rust cold starts are almost completely CPU bound, so you see the near perfect scaling there. Meanwhile Python I'd guess is mostly io bound, which doesn't scale. That kinda makes sense as Go/Rust compile down to single executable while Python loads lots of libraries from disk. For Rust, I suspect spinning up Tokio and the rest of async runtime might be making cold starts worse. But that's purely speculation. The python lambda is good old-fashioned sync io. Another difference is that Rust lambda initializes a logger, while Python one doesn't. That might add some milliseconds too to the startup.

kidsil 5y ago

I find it surprising that NodeJS is showing the worst cold start times. Most likely over 90% of AWS Lambda functions are written NodeJS.

tyingq 5y ago

Nice to have some updated data and comparisons. This article doesn't include the effect if the Lambda has to connect to a VPC though, which adds time for the ENI. Though that was greatly improved in 2019-2020: https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/ https://aws.amazon.com/blogs/compute/announcing-improved-vpc...

_fat_santa 5y ago

At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint for the first time, it would take 2x as long as it normally would at first. To combat this we wrote a "runWarm" function that kept the API alive at all times. Sure kind of defeats the purpose of serverless but hey, enterprise software.

devonkim 5y ago
To be fair, all you’d need to accomplish that without more than necessary parts for production is to ensure that the code path invoking the function is accessed via an external monitoring probe with an adjustment to SLA or SLO to account for the cold start time. Obviously not going to work for many systems, but it’s easy to forget all the side effects of the observability plane when writing applications.
deleted 5y ago
[deleted]
nodejs_rulez_1 5y ago
> ...Sure kind of defeats the purpose of serverless but hey, enterprise software. No, no, no. It's "hey, Amazon/Microsoft cloud engineering". They should be amazing with whiteboard interview exercises though.
jpdb 5y ago
> To combat this we wrote a "runWarm" function that kept the API alive at all times. This doesn't really work like you'd expect and isn't recommended, as it only helps a particular use-case. The reason is that AWS Lambda will only keep a single instance of your function alive. That means if two requests come in at the same time, you'd see a cold start on one of those invocations. Instead, you want to look at something like provisioned concurrency.
- VWWHFSfQ 5y ago
  Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely better. I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.

jabart 5y ago

We run a few .net core lambdas and a few things that make a big difference for latency. 1. pre-jit the package, this reduces cold start times as the JIT doesn't need to run on most items. Still does later to optimize some items. 2 is sticking to the new .net json seralizer. The reference code uses both the new and old newtsonsoft package. The old package has higher memory allocations as it doesn't make use of the Span type.

moonchrome 5y ago
Great tip for running the JIT AOT, for anyone interested Microsoft calls this "ReadyToRun" compilation [1]. [1] https://docs.microsoft.com/en-us/dotnet/core/deploying/ready-to-run https://docs.microsoft.com/en-us/dotnet/core/deploying/ready... I wonder did you test if the increased size results in an actual win for the startup time ?
- jabart 5y ago
  After like 256mb it is less of an impact when using readytorun. Some of the lambdas I have are webhooks so latency isn't as important and when it's user facing 512mb seems to be a sweet spot.
- Shadonototra 5y ago
  and then you need to upgrade the plan because it is disk/memory hungry and doesn't fit within the 128mb plan, so you'll pay 4x more already just use GO and be faster while saving money at the same time
  - moonchrome 5y ago
    Like someone else said lambdas are priced by memory * execution time - if you cut execution time by 1/2 by doubling the memory and using AOT - you got faster lambdas for free (even if it's not as high as 1/2 you'll probably still not be paying x2).
- gokhan 5y ago
  The test code is quite small and might not benefit from R2R that much, libs it relies on are already jitted. Ditching Newtonsoft would affect response time though.

unraveller 5y ago

just the cold-start data: https://gist.github.com/Aleksandr-Filichkin/925fce9d910e04d2037f18caf3cfc3a5/raw/44d39a5091c822aa1d653246413cf06b404adc01/Cold-start..tsv https://gist.github.com/Aleksandr-Filichkin/925fce9d910e04d2... mirror: https://scribe.rip/@filia-aleks/aws-lambda-battle-2021-performance-comparison-for-all-languages-c1b441005fd1 https://scribe.rip/@filia-aleks/aws-lambda-battle-2021-perfo...

moonchrome 5y ago

One thing I wish he included was .NET 5 - since that switched to using Docker images I would be very interested in the differences.

ewjt 5y ago
Lambda only has default support for the LTS releases of .NET, which is 2.1, 3.1 and the upcoming (November 2021) .NET 6. The only way to run .NET 5 in a Lambda that I know of would be custom runtimes or containers. Is that what do you mean by “switched to using Docker images”? This article describes the containerized performance of .NET in Lambda and the cold starts are dramatically (~4x) worse. https://www.kloia.com/blog/aws-lambda-container-image-.net-benchmark https://www.kloia.com/blog/aws-lambda-container-image-.net-b...
- moonchrome 5y ago
  >Lambda only has default support for the LTS releases of .NET Not true, https://aws.amazon.com/blogs/developer/net-5-aws-lambda-support-with-container-images/ https://aws.amazon.com/blogs/developer/net-5-aws-lambda-supp... >Even though Lambda’s policy has always been to support LTS versions of language runtimes for managed runtimes, the new container image support makes .NET 5 a first class platform for Lambda functions.

sp33der89 5y ago

I would love to see languages like OCaml, D, Nim benchmarked here as well. They sit sortof in between Go and Rust, where I don't have to deal with manual memory management but get enough expressiveness to write a nice Lambda.

ignoramous 5y ago
Those, and: Zig would provide for an interesting contrast with Rust and Nim.
throwaway894345 5y ago
Not sure about the others, but OCaml tends to perform in the same ballpark as Go, and a bit slower for programs that benefit from shared memory parallelism.

cloudytechi147 5y ago

Many organizations use Linux OS to run file servers, print servers, content delivery systems, global caching servers, data archives, VPN servers, etc. Surely, Windows and macOS are easy to use, but Linux distributions have become more classy and user-friendly over the years. Linux is also considered to be more secure than Windows and macOS and makes software deployment quite easy. There are certain Linux distros that support enterprise-related tasks, and for that, you can consider these options: Blog link: https://webhostingprime.com/best-linux-os/ https://webhostingprime.com/best-linux-os/

psanford 5y ago

Something I discovered recently, for my tiny Go Lambda functions it is basically always worth it to run them at least with 256mb of memory even if they don't need more than 128mb. This is because most of my functions run twice as fast at 256mb than they do at 128mb. Since lambda pricing is memory_limit times execution time, you get better performance for free. Test your lambda functions in different configurations to see if the optimal setting is different than the minimal setting.

jSherz 5y ago
CPU allocated to Lambda functions scales with the memory allowance [1] and you can use power tuning [2] to find the optimal values. [1] https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html https://docs.aws.amazon.com/lambda/latest/dg/configuration-f... [2] https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:451282441545:applications~aws-lambda-power-tuning https://serverlessrepo.aws.amazon.com/applications/arn:aws:s...

xwdv 5y ago

If cold start times are an issue use Cloudflare workers instead, they’re always warm.

Jonovono 5y ago
oh baby, they are warm indeed

cmcconomy 5y ago

My experience with cold starts in Azure Functions Serverless is pretty awful. Like most other Azure services, their affordable consumer grade offerings are designed from the ground up not to be good enough for "serious" use. Cold start times compared to Lambda are worse, and in addition, we would get random 404s which do not appear in any logs; inspecting these 404s indicated they were emitted by nginx, leading me to believe that the ultimate container endpoint was killed for whatever reason but that fact didn't make it back to the router, which attempted and failed to reach the function. Of course the cold start and 404 are mitigated if you pay for the premium serverless or just host their middleware on their own App Service plans (basically VMs)

secondaryacct 5y ago
My company is moving away from datacenter and into Azure and I have to get the az900 this week, it doesnt bode well. And I was so happy to leave the clusterfuck of 300 aws lambda I was working with in my prev company. What an expensive fad, and no engineer is ever consulted ...
- cmcconomy 5y ago
  There are things I like, such as the consistent and pervasive security model with Azure AD. I don't even mind running stuff on App Services. I just find their commodity serverless offering in particular is subpar
  - meekins 5y ago
    > consistent and pervasive security model with Azure AD Wait, this is the first time I hear this about Azure. Could you elaborate? It is possible that things have improved significantly since I last worked with Azure but lack of a consistent security model (like IAM on AWS) to control human and service (Azure Functions, App Service apps etc) access to specific resources (Cosmos databases, EventHubs etc) especially painful.
    - cmcconomy 5y ago

flurie 5y ago

I would have liked to see more values along the lambda "breakpoints" between 1GB and 10GB of memory. Unless things have changed recently, my understanding is that CPU and IO scale up specifically at those breakpoints rather than being continuous.

losvedir 5y ago

I'm surprised Node has cold-start issues. I had it in my mind that JS was Lambda's "native" language and wouldn't have cold start issues at all. Did it used to be like that? Didn't Lambda launch with only support for JS, and maybe a couple other languages that could compile to it?

yoava 5y ago
With node.js, the cold start problem is caused by how node loads files. For each file it does about 10 IO operations (to resolve the file from the module name), then load, parse and compile the file. If using any file system that is not super fast, this amounts to long delays. There are ways to get around that, but those are not available on lambda
maxmcd 5y ago
I thought nodejs/v8 or any javascript runtime would have some kind of startup cost since it has to parse and compile the javascript code first. See a simple hello world execution time comparison: # a Go hello world $ time ./hello hi real 0m0.002s $ time echo 'console.log("hello")' | node - hello real 0m0.039s The ~25ms of cold start noted in this article feels acceptable and impressive to me, given what node is doing under the hood.
artimaeis 5y ago
Yeah, at launch Lambda only supported Node.js. https://web.archive.org/web/20141115183837/http://aws.amazon.com/lambda/faqs/ https://web.archive.org/web/20141115183837/http://aws.amazon...
seniorsassycat 5y ago
I wonder ho w much time was spent requiring all of aws-sdk. The v3 sdk is modular and should be quicker to load. Bundlers like rebuild save space and reduce parsing time.
- lukeramsden 5y ago
  Double digit ms in my experience. Not including the fact that V2 will renegotiate TLS constantly when it could just keep the socket open.

_wldu 5y ago

It's really amazing to see how Lambda has grown to support all of these languages.

haolez 5y ago

Slightly off topic, but what's the deal with Azure Functions cold start times in the Consumption (i.e. serverless) plan? I get cold start times in the multi seconds range (sometimes huge values, like 20s). Am I doing something wrong? Or is this expected?

kkielhofner 5y ago
I've experienced this as well. I gave up on optimizing it. I get around it by using a load balancer (Cloudflare currently) that does periodic health checks. Keeps it alive and the charges are minimal (still well within the free tier). Speaking of Cloudflare it's on my "nice to-do" list to move this to Workers as a primary anyway. I also use a completely separate uptime monitor and alerting platform (Uptime Robot) so one way or another I'd be keeping at least one instance warm no matter what.

Shadonototra 5y ago

that is why the language for cloud native is GO, and languages like Java/C# are dead tech stack, they failed to reinvent themselves to stay relevant

FpUser 5y ago
>"they failed to reinvent themselves to stay relevant" This sounds like it came straight out of "The Corporate BS generator" - https://www.atrixnet.com/bs-generator.html https://www.atrixnet.com/bs-generator.html.
- Shadonototra 5y ago
  no, it's the direct translation of the benchmark of this post cold start should be a thing of the past, hence the "failed to reinvent themselves to stay relevant" you need to speak to the hardware directly, not to the VM or the embedded compiler with ARM and ultimately RISC-V coming, that's gonna be even more true
  - ewjt 5y ago
    How does a single metric from a highly specialized runtime environment indicate a tech stack is dead? There are things you can do right now [1] to mitigate these cold start issues. Going forward, ahead-of-time compilation will be an option. [2] Aside from cold starts, note that the improvements in .NET make ASP.NET Core one of the fastest web frameworks. [3] The article: > “.Net has almost the same performance as Golang and Rust, but only after 1k iterations(after JIT).” Additions like async/await and nullable reference types make it easier to write bug-free code, which for a lot of folks is a better trade off than “speaking to the hardware directly”. .NET also runs natively on a bunch of platforms now, including ARM. I’d call all of that continuous improvement. Perhaps even reinvention? [1] https://docs.microsoft.com/en-us/dotnet/core/deploying/ready-to-run https://docs.microsoft.com/en-us/dotnet/core/deploying/ready... [2] https://github.com/dotnet/runtimelab/tree/feature/NativeAOT https://github.com/dotnet/runtimelab/tree/feature/NativeAOT [3] https://www.techempower.com/benchmarks/#section=test&runid=57b25c85-082a-4013-b572-b0939006eaff&hw=ph&test=composite&a=2 https://www.techempower.com/benchmarks/#section=test&runid=5...

projectileboy 5y ago

AWS Lambda is pretty cool, it just gets used a lot for applications that it was never really designed for. While I wish that Amazon would address the cold start times, if you try to grill your burgers with a cordless drill, you can’t really blame the drill manufacturer when the meat doesn’t cook.

rcarmo 5y ago

I recently discovered that uWSGI has a "cheap mode" that will hold the socket open but only actually spawn workers when a connection comes in (and kill them automatically after a timeout without any requests). Pertinent options: https://github.com/piku/piku/blob/master/piku.py#L908 https://github.com/piku/piku/blob/master/piku.py#L908 If you already have 24/7 compute instances going and can spare the CPU/RAM headroom, you can co-host your "lambdas" there, and make them even cheaper :)

tinyprojects 5y ago

If anyone is running into cold start problems on Firebase, I recently discovered you can add .runWith({minInstances: 1}) to your cloud functions. It keeps 1 instance running at all times, and for the most part completely gets rid of cold starts. You have to pay a small cost each month (a few dollars), but its worth it on valuable functions that result in conversions, e.g. loading a Stripe checkout.

digianarchist 5y ago
This is basically how Google Cloud Run works if I'm not mistaken.
- fulafel 5y ago
  And AWS Lambda "provisioned concurrency".
- prattmic 5y ago
  Google Cloud Run also scales to zero by default, but you can configure the minimum number of instances: https://cloud.google.com/run/docs/configuring/min-instances https://cloud.google.com/run/docs/configuring/min-instances

fulafel 5y ago

Container based lambda image configurations (vs zip based) would be a good addition to this comparison. People use them eg to get over the zip based lambda size limit. Also maybe mentione provisioned concurrency (where you pay AWS to keep one or more instances of your lambda warm). Both of these are supported by Serverless framework btw.

time0ut 5y ago
Definitely. In my experience, docker image based Lambdas had consistently poor (>3s) cold starts regardless of memory. I hope it will eventually improve as it is a much nicer packaging approach than ZIP file. Also, it would have been nice to include ARM vs x86 now that ARM is available.

e3bc54b2 5y ago

I just want to appreciate the article. Starting with non-clickbait title, upfront summary, detailed numbers, code for reruns, great graphs, no dreamy story and no advertisement of any kind. It is hosted on Medium but the author has done a banging great job, so gets a pass. If he is reading, excellent work!

gunnarmorling 5y ago

The best cold starts are those which aren't noticed by the user. For my blog search (which runs on Lambda), I found a nice way of achieving that [1]: as soon as a user puts the focus to the input field for the search text, this will already submit a "ping" request to Lambda. Then, when they submit the actual query itself, they will hit the already running Lambda most of the times. And, as others said, assigning more RAM to your Lambda than it actually may need itself, will also help with cold start times, as this increases the assigned CPU shares, too. [1] https://www.morling.dev/blog/how-i-built-a-serverless-search-for-my-blog/ https://www.morling.dev/blog/how-i-built-a-serverless-search...

buzzdenver 5y ago
So effectively you're going to pay double?
- musingsole 5y ago
  You increase the usage of the lambda, sure, but not by double unless all this lambda does is respond to the search. There's a point where this is all semantics when running cloud services with regard to how you've deployed your code behind gateways and lambdas, but most applications have obvious triggers that could warm up a lambda at 10% or less overhead. And even then....lambdas are dirt cheap.
  - wpietri 5y ago
    Lambdas can be dirt cheap if you use them well. But that's always what happens. I recently saw a report about contractor who used them for crawling websites. Their client saw a surprise $12k bill for Lambda, when a simple Scrapy crawler on a low-end instance would have cost them < $100/month for the same load. Why? Because if you spin up a ton of Lambda invocations and have them sit around waiting for the network, you pay for each CPU to sit idle, rather than having just one CPU stay busy managing a bunch of async IO.
    - laurencerowe 5y ago

davewritescode 5y ago

The main downside of Lambda, in particular for user facing applications is that the incentives of the cloud provider and you are completely opposed. You (the developer) want a bunch of warm lambdas ready to serve user requests and the cloud provider is looking to minimize costs by keeping the number of running lambdas as low as possible. It's the incentive model that fundamentally makes Lambda a poor choice for these types of applications. Other downsides include the fact that Lambdas have fixed memory sizes. If you have units of work that vary in amount of memory required you're basically stuck paying the costs of the largest units of work unless you can implement some sort of routing logic somewhere else. My company ran into this issue using lambdas to process some data where the 99% of requests were fine running in 256mb but a few required more. There was so way to know ahead of time how much memory the computation would require ahead of time. We ended up finding a way to deal with it but in the short term we had to bump the lambda memory limits. That doesn't even get into the problems with testing. In my experience, Lambdas are best used as glue between AWS components, message processors and cron style tasks.

paxys 5y ago
Disagree with "completely opposed". Cloud providers want to make money, sure, but in general everyone in the ecosystem benefits if every CPU cycle is used efficiently. Any overhead goes out of both AWS's and your pockets and instead to the electricity provider, server manufacturer, cooling service.
joncrane 5y ago
> the incentives of the cloud provider and you are completely opposed I think this is a little overstated. The cloud provider wants their customers to be happy while minimizing costs (and therefore costs to the customer). It's not truly a perverse incentive scenario.
tuananh 5y ago
> In my experience, Lambdas are best used as glue between AWS components, message processors and cron style tasks. i think lambda is the defacto standard now for integration on aws. but they are pushing lambda hard for other workload too

l8again 5y ago

I wish the author had done a comparison for Java apps using JLink that generates a custom Java runtime image that contains only the platform modules that are required for a given application, and if that makes a difference.

gunnarmorling 5y ago
jlink usage won't make any significant difference typically. What does make a difference is (App)CDS though, as available in newer Java versions. Memory-mapping a file with the class metadata from previous runs can easily shave off a second or more from time-to-first-response [1], depending on number and size of classes required for that. [1] https://www.morling.dev/blog/smaller-faster-starting-container-images-with-jlink-and-appcds/ https://www.morling.dev/blog/smaller-faster-starting-contain...

Kalanos 5y ago

i read that they offer C as well. wonder what those times look like

eldavido 5y ago
It's low (100-200ms). Read my post above

eldavido 5y ago

C++. 200ms cold start. provided.al2 runtime environment. Lambda success story: Started with a .NET Core API about a year ago. Monolith-first. Mix of clients across mobile and React. Async/await is one of the better things about C# (the language used for ASP.NET Core) and as a result, we were able to do things you'd never consider doing in-process on a system like Ruby on Rails (right on the thread serving the HTTP request), like transcoding a 12 megapixel HEIC upload into JPEG. We just did it, left the connection open, and when it was done, returned an HTTP 200 OK. That worked well for a while and let us serve tons of clients on a single Heroku dyno. The problem: memory. Resizing images takes tens/hundreds of MB when you're doing it into three different formats. Over the last two weeks, I extracted the HEIC->JPEG transcode/resize out of our monolith into a Lambda. I'm extremely happy with how it turned out. We went with C++ because the whole idea was performance, we're going to be doing point cloud processing and other heavyweight stuff, and wanted fine-grained control of memory. Our process has 28MB of dynamic libraries (.so files), starts in 200ms, and runs comfortably on a 512MB instance. We moved to 1024 to provide a margin of safety just in case we get a really large image. The system has progressed into "I don't even think about it"-level reliably. It just works and I pay something like $1 for 40-50k transcode operations. No EC2 instances to manage, no queues, no task runners, no Ruby OOM, no running RabbitMQ, none of that (former ops engineer at a very high-scale analytics company). As a general comment, I don't see many cloud services written in C/C++. This is no doubt partly because those skills just aren't widespread. But I think the bigger lesson is that it might be worth adding a little bit of development complexity to save 10x as much ops overhead. When I explained this setup to my friend, his first reaction was, "Why didn't you just put ImageMagick (the binary) into a container?" Once I explained that actually, I need to get images from S3, and write them into several formats, and manipulate their S3 keys in somewhat complex ways, fire off an HTTP request to a server, and pass a JWT around...sure, I could write this in a shell script, with wget, and curl, and everthing else. But at some point you just have to write the right code for the job using the right tools. I think hybrid approaches like this make the most sense. .NET and Java are great high-productivity tools for running server apps where memory is relatively abundant. I wouldn't try to move a system like that onto Lambda any more than I'd try to do something that more naturally fits with a queue/worker pattern on a webserver. This seems kind of obvious but if I'm being honest, it's probably experience talking a bit. It's also neat to just get back to the metal a bit. Drop the containers, runtime environments, multi-hundred-MB deployment packages, just ship a xx MB package up to the cloud, deploy it, and have it run as a standalone linux binary with all the speed and simplicity that brings. Modern C++ is a totally different animal than 90s C++, I'd encourage giving it a try if you haven't in a while.

NicoJuicy 5y ago

Any experience with Cloudflare workers? They are supposed to have 0 cold start time: https://blog.cloudflare.com/eliminating-cold-starts-with-cloudflare-workers/ https://blog.cloudflare.com/eliminating-cold-starts-with-clo...

erikerikson 5y ago

I was surprised by the quality of this one. That said... Cold starts are a FaaS learning subject but they almost never matter much in practice. What workloads are intermittent and also need extremely low latencies? Usually when I see people worrying about this it is because they have architected their system with call chains and the use case, if it really matters, can be re-architected so that the query result is pre prepared. This is much like search results... Search engines certainly don't process the entire web to service your queries. Instead, they pre-calculate the result for each query and update those results as content CRUD happens.

somethingAlex 5y ago

Honestly shocked that rust is about 4 times faster than node for the DynamoDB insert workload in the average case. I knew it'd be faster but I would have expected maybe 50% faster since most of the time is probably spent simply sending the data to dynamoDB and awaiting a response. Also, what is up with Python being faster than Node in the beginning and then getting slower over time? The other languages (apart from graal) get faster over time. I'm referring to the average MS latency at 128MB graph at the bottom.

deleted 5y ago
[deleted]

Havoc 5y ago

Where feasible I’m using cloudflare workers (and KV) instead to be honest. Less versatile but no tangible cold start time

yoava 5y ago

The article states that 600ms is a low cold start figure. However, 600ms cold start is still unacceptable for usage by WebApps. For webapp, the figure should be 100ms. The only platform that meets that figure (that I know of) is Velo by Wix with around 50ms cold start for node.js

bezossucks 5y ago

If you can run your entire function in V8 and don't need node, Cloudflare Workers is MUCH faster, more affordable, more manageable, and more reliable. They get cloned all over the world and you're not region-locked. https://www.cloudflare.com/learning/serverless/serverless-performance/ https://www.cloudflare.com/learning/serverless/serverless-pe...

wertgklrgh 5y ago
yet nobody ever uses them
StanAngeloff 5y ago
We've moved from AWS to Cloudflare and I can confirm Workers are awesome. The only downside is lots of work required to get npm packages running properly. Due to the lack of Node.js' runtime, bundling, shims and stubbing required to get stuff like "fs" and "net" compiling. AWS is more mature, has a better community support around it and generally the tooling is more stable. Workers are quickly catching up, though, with great new additions such as ES modules, custom builds, Miniflare, etc.

rimutaka 5y ago

The articles states that it takes approximately 1000 requests to optimize / warm up. I suspect that they had concurrency set at the default 999, so the first 999 requests would spin up new instances. Does that mean their 15,000 requests were actually 15 requests spread over 1000 instances?

The_rationalist 5y ago

It's unfortunate than ZGC and Shenandoah weren't tested nor parallel gc

asdev 5y ago

This guy has such a rudimentary understanding that he can't point to real examples of Ethereum applications that support his point. Most smart contracts are immutable and non-upgradeable, which nullifies this entire blog post. Many protocols are also moving to DAO governance for even further decentralization of power. This reflects a highly pervasive issue as people with a somewhat technical background but no experience with crypto networks and applications think they are qualified to give opinions on whether the protocols will make it or not.

mulmboy 5y ago

A pattern I have implemented is to have my API code on both ECS/Fargate and Lambda at the same time, and send traffic to the appropriate one using an Elastic Load Balancer. I flag specific endpoints as "cpu intensive" and have them run on lambda. Implemented by - Duplicating all routes in the API with the "/sls/" prefix (this is a couple of lines in FastAPI) - Setting up a rule in ELB to route to Lambda if the route starts with /sls, or to ECS otherwise. - Set up the CPU intensive routes to automatically respond with a 307 to the same route but prefixed with /sls. Boom, with that the system can handle bursts of CPU intensive traffic (e.g. data exports) while remaining responsive to the simple 99% of requests all on one vCPU. And the same dockerfile, with just a tiny change, can be used both in ECS and Lambda.

tiew9Vii 5y ago

The Rust metrics are interesting. I've been getting around 20ms cold starts, 1ms warm exec on the 128MB ARM Graviton2 using Rust for the most basic test cases. Graviton2 was slightly slower on cold starts than X86 for me (1-2ms) but who doesn't want to save $0.0000000004 per execution? Adding calls to parameter store/dynamo DB bumps it up a little but still < 120ms cold, and any added latency comes from waiting on the external service calls. Memory usage is 20-30MB, and I haven't done anything to optimise memory. I know I can get rid of a few allocations I'm doing for simplicity if I want to. I've not always been the greatest fan of Lambdas seeing it has hidden complexity orchestrating and a blackbox for debugging. Re-visiting a few years on and with Rust, you get an excellent language, excellent runtime characteristics and substantial cost savings unless you really need more than 128MB memory, i.e. processing large volumes of data per execution in memory or transcoding. Any asynchronous/event-driven service I write, I'll just package as a Rust lambda going forward and pay fractions of a cent per month. I am still on-the-wall with HTTP exposed services as that's a big plumbing exercise and hidden gateway costs but not as adverse to it as I was.

bilalq 5y ago

> NodeJs is the slowest runtime, after some time it becomes better(JIT?) but still is not good enough. In addition, we see the NodeJS has the worst maximum duration. The conclusion drawn about NodeJS performance is flawed due to a quirk of the default settings in the AWS SDK for JS compared to other languages. By default, it opens and closes a TCP connection for each request. That overhead can be greater than the time actually needed to interact with DDB. I submitted a pull request to fix that configuration[0]. I expect the performance of NodeJS warm starts to look quite a bit better after that. [0]: https://github.com/Aleksandr-Filichkin/aws-lambda-runtimes-performance/pull/6 https://github.com/Aleksandr-Filichkin/aws-lambda-runtimes-p...

bilalq 5y ago
In addition, the NodeJS cold start time can be further optimized by bundling into a single file artifact to reduce the amount of disk IO needed when requiring dependencies. Webpack, Parcel, ESBuild, and other bundlers could achieve that, I'm sure. EDIT: That may already be happening here in the build.sh file. I see it runs `sam build --use-container NodeJsFunction -b nodejs`.

paulmendoza 5y ago

I wonder if he used the special .net compilation setting that reduces quick starts. It requires compiling the function on Amazon Linux 2