5 ms·

I am curious to see if this will mean a shift to more efficient languages (Go or Rust) for Lambda services, as usually people default to JS

tgv 6y ago
I've got bad experiences with go startup (i.e., cold runs). They're much more expensive than I would have expected. If node can indeed run in 40ms (as https://news.ycombinator.com/item?id=25267211 https://news.ycombinator.com/item?id=25267211 says), then I'm surely going back to JS.
- philosopher1234 6y ago
  What is your experience? Go is an AOT compiled language so the only thing I could imagine you running into on startup is loading the binary into memory? Theres not a cold-start issue with Go, as its not an optimizing JIT. Edit: Bizarre. Seems like Go on lambda does actually have slower cold start than JS or Python. I wonder if its just that the binary is likely larger than the equivalent JS source code? https://levelup.gitconnected.com/aws-lambda-cold-start-language-comparisons-2019-edition-%EF%B8%8F-1946d32a0244 https://levelup.gitconnected.com/aws-lambda-cold-start-langu...
  - nicoburns 6y ago
    I'm not sure about lambda, but cloudflare workers use v8 isolates to avoid starting a new process at all.
  - tgv 6y ago
    My experience is that Go cold start takes around 900ms. Processing (parsing a JSON message, verifying it with one DynamoDB look-up, and storing it in DynamoDB) then takes between 11ms and 105ms. Go does use less memory than node, though, and that also counts in Lambda. I hadn't expected it either, but it loads node faster. Perhaps via some VM trick?
  - gen220 6y ago
    > The function did nothing except emit ‘hello world’. A more realistic benchmark would be parsing a 1kb protobuf blob and printing some random key from it. (this would require importing a non-stdlib parser) Without knowing how it's implemented, my guess is that they're conserving python/v8 processes, so that they're not cold-starting the interpreter on each lambda execution. You can't [1] do the same thing for a Go binary, so they have to invoke a binary, which might involve running some scans against it first. This leads to some pretty counterintuitive conclusions! If you want minimal latency (in Lambda!!), you really should be using JS/Python, I guess. [1]: OK. Maybe you could. Go has a runtime after all, although it's compiled into the binary! I have never heard of anybody doing this, but I'd love to read something about it. :)
  - indymike 6y ago
    Go binaries are pretty large compared to say, C.
  - Thaxll 6y ago
    Go is not AOT compiled.
    - philosopher1234 6y ago
      ... yes it is?
- gabipurcaru 6y ago
  same. I went through the trouble of implementing my function in Rust (Rocket), and it's actually quite slow because (a) startup is slow and (b) async/await is still pretty painful to use so I'm blocking on IO
- Tehnix 6y ago
  I wrote this in another comment here, but just so you don't miss it: Don't. Dependencies for the dynamic languages matter A LOT! Take a look at what it'll cost you for requiring the AWS SDK in Node.js, for your cold starts https://theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/ https://theburningmonk.com/2019/03/just-how-expensive-is-the... Personal benchmarks puts Rust as the most optimal language that I've tried to run on AWS Lambda so far.
cle 6y ago
IME Lambda functions are mostly sitting around waiting on I/O, so I don't think it would make much of a difference for those workloads. The important technical factors for those workloads are startup time and I/O capabilities...JS is strong in both of those areas. For simple Lambda functions JS still seems like a great choice, along with Go. Rust would be overkill IMO unless you need to share a codebase or aren't I/O bound or have some other unique requirements.
k__ 6y ago
On the one hand I read that JS Lambdas were often already under 100ms (30-50ms) On the other hand I heard legends about under 10ms Rust Lambdas.
- wgjordan 6y ago
  That's the point- billed per ms, a Lambda that executes in 5ms is 10x cheaper than one that takes 50ms. Billed per 100ms interval, the total cost of the two is the same.
  - k__ 6y ago
    Yes, I'm not questioning that 5 is a tenth of 50. I'm questioning the Rust speed :D
Swizec 6y ago
JS is a great choice for Lambda thanks to great cold performance. I’m seeing runtimes in the 40ms to 100ms range. Most of the time in Lambda is usually spent waiting for IO, which is slow in any language. If you’re using Lambda for heavy computation, that’s not a great choice.
- tnolet 6y ago
  Yep, we ran ~50M Node invocations last month on a small function. AVG was around 100ms but lots of sub 50ms invocations too.
valbaca 6y ago
From the research I did, here's how languages stack up in Lambda runtime (lowest first): 1. Python & JS 2. Go 3. C# & Java I couldn't find any data on Rust. The understanding at the time was that Python & JS runtimes are built-in, so the interpreter is "already running" Go is the fastest of compiled languages, but just can't beat the built-in runtimes. C# and Java were poorest as they're spinning up a larger runtime that's more optimized for long-running throughput. https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html https://docs.aws.amazon.com/lambda/latest/dg/best-practices.... https://medium.com/the-theam-journey/benchmarking-aws-lambda-runtimes-in-2019-part-i-b1ee459a293d https://medium.com/the-theam-journey/benchmarking-aws-lambda... https://epsagon.com/development/aws-lambda-programming-language-comparison/ https://epsagon.com/development/aws-lambda-programming-langu... https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581 https://read.acloud.guru/comparing-aws-lambda-performance-of... Of course, benchmarks like this only go so far. Use as a starting point for your own evaluation; not as an end-all-be-all.
- throwaway894345 6y ago
  I’m not sure I’m interested in a hello world benchmark if it takes Python 5 seconds to import its dependencies in the real world.
  - stefano 6y ago
    This is very true. Just importing Django + Django Rest Framework + some other minor libraries in Google App Engine (standard) leads to painfully slow response times when a new instance spins up. Like, more than 10s to spin up an instance. Although App Engine seems to be 3-4 times slower than my desktop computer from 2014 on this particular task. I wonder if AWS lambda is better.
  - Tehnix 6y ago
    Dependencies for the dynamic languages matter A LOT! Take a look at what it'll cost you for requiring the AWS SDK in Node.js, for your cold starts https://theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/ https://theburningmonk.com/2019/03/just-how-expensive-is-the....
munns 6y ago
A lot of this was based around the fact that we've seen languages become just so much more performant. This includes Go/Rust/etc, but a lot of Node.js workloads are also sub 100ms, or fast enough that they'd benefit from this pretty well. - Chris, Serverless@AWS
JJJollyjim 6y ago
I just checked cloudwatch for my rust-based function, I'm now being billed 1ms :D

tgv 6y ago

I've got bad experiences with go startup (i.e., cold runs). They're much more expensive than I would have expected. If node can indeed run in 40ms (as https://news.ycombinator.com/item?id=25267211 https://news.ycombinator.com/item?id=25267211 says), then I'm surely going back to JS.

philosopher1234 6y ago
What is your experience? Go is an AOT compiled language so the only thing I could imagine you running into on startup is loading the binary into memory? Theres not a cold-start issue with Go, as its not an optimizing JIT. Edit: Bizarre. Seems like Go on lambda does actually have slower cold start than JS or Python. I wonder if its just that the binary is likely larger than the equivalent JS source code? https://levelup.gitconnected.com/aws-lambda-cold-start-language-comparisons-2019-edition-%EF%B8%8F-1946d32a0244 https://levelup.gitconnected.com/aws-lambda-cold-start-langu...
- nicoburns 6y ago
  I'm not sure about lambda, but cloudflare workers use v8 isolates to avoid starting a new process at all.
- tgv 6y ago
  My experience is that Go cold start takes around 900ms. Processing (parsing a JSON message, verifying it with one DynamoDB look-up, and storing it in DynamoDB) then takes between 11ms and 105ms. Go does use less memory than node, though, and that also counts in Lambda. I hadn't expected it either, but it loads node faster. Perhaps via some VM trick?
- gen220 6y ago
  > The function did nothing except emit ‘hello world’. A more realistic benchmark would be parsing a 1kb protobuf blob and printing some random key from it. (this would require importing a non-stdlib parser) Without knowing how it's implemented, my guess is that they're conserving python/v8 processes, so that they're not cold-starting the interpreter on each lambda execution. You can't [1] do the same thing for a Go binary, so they have to invoke a binary, which might involve running some scans against it first. This leads to some pretty counterintuitive conclusions! If you want minimal latency (in Lambda!!), you really should be using JS/Python, I guess. [1]: OK. Maybe you could. Go has a runtime after all, although it's compiled into the binary! I have never heard of anybody doing this, but I'd love to read something about it. :)
- indymike 6y ago
  Go binaries are pretty large compared to say, C.
- Thaxll 6y ago
  Go is not AOT compiled.
  - philosopher1234 6y ago
    ... yes it is?
gabipurcaru 6y ago
same. I went through the trouble of implementing my function in Rust (Rocket), and it's actually quite slow because (a) startup is slow and (b) async/await is still pretty painful to use so I'm blocking on IO
Tehnix 6y ago
I wrote this in another comment here, but just so you don't miss it: Don't. Dependencies for the dynamic languages matter A LOT! Take a look at what it'll cost you for requiring the AWS SDK in Node.js, for your cold starts https://theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/ https://theburningmonk.com/2019/03/just-how-expensive-is-the... Personal benchmarks puts Rust as the most optimal language that I've tried to run on AWS Lambda so far.

cle 6y ago

IME Lambda functions are mostly sitting around waiting on I/O, so I don't think it would make much of a difference for those workloads. The important technical factors for those workloads are startup time and I/O capabilities...JS is strong in both of those areas. For simple Lambda functions JS still seems like a great choice, along with Go. Rust would be overkill IMO unless you need to share a codebase or aren't I/O bound or have some other unique requirements.

k__ 6y ago

On the one hand I read that JS Lambdas were often already under 100ms (30-50ms) On the other hand I heard legends about under 10ms Rust Lambdas.

wgjordan 6y ago
That's the point- billed per ms, a Lambda that executes in 5ms is 10x cheaper than one that takes 50ms. Billed per 100ms interval, the total cost of the two is the same.
- k__ 6y ago
  Yes, I'm not questioning that 5 is a tenth of 50. I'm questioning the Rust speed :D

Swizec 6y ago

JS is a great choice for Lambda thanks to great cold performance. I’m seeing runtimes in the 40ms to 100ms range. Most of the time in Lambda is usually spent waiting for IO, which is slow in any language. If you’re using Lambda for heavy computation, that’s not a great choice.

tnolet 6y ago
Yep, we ran ~50M Node invocations last month on a small function. AVG was around 100ms but lots of sub 50ms invocations too.

valbaca 6y ago

From the research I did, here's how languages stack up in Lambda runtime (lowest first): 1. Python & JS 2. Go 3. C# & Java I couldn't find any data on Rust. The understanding at the time was that Python & JS runtimes are built-in, so the interpreter is "already running" Go is the fastest of compiled languages, but just can't beat the built-in runtimes. C# and Java were poorest as they're spinning up a larger runtime that's more optimized for long-running throughput. https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html https://docs.aws.amazon.com/lambda/latest/dg/best-practices.... https://medium.com/the-theam-journey/benchmarking-aws-lambda-runtimes-in-2019-part-i-b1ee459a293d https://medium.com/the-theam-journey/benchmarking-aws-lambda... https://epsagon.com/development/aws-lambda-programming-language-comparison/ https://epsagon.com/development/aws-lambda-programming-langu... https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581 https://read.acloud.guru/comparing-aws-lambda-performance-of... Of course, benchmarks like this only go so far. Use as a starting point for your own evaluation; not as an end-all-be-all.

throwaway894345 6y ago
I’m not sure I’m interested in a hello world benchmark if it takes Python 5 seconds to import its dependencies in the real world.
- stefano 6y ago
  This is very true. Just importing Django + Django Rest Framework + some other minor libraries in Google App Engine (standard) leads to painfully slow response times when a new instance spins up. Like, more than 10s to spin up an instance. Although App Engine seems to be 3-4 times slower than my desktop computer from 2014 on this particular task. I wonder if AWS lambda is better.
- Tehnix 6y ago
  Dependencies for the dynamic languages matter A LOT! Take a look at what it'll cost you for requiring the AWS SDK in Node.js, for your cold starts https://theburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/ https://theburningmonk.com/2019/03/just-how-expensive-is-the....

munns 6y ago

A lot of this was based around the fact that we've seen languages become just so much more performant. This includes Go/Rust/etc, but a lot of Node.js workloads are also sub 100ms, or fast enough that they'd benefit from this pretty well. - Chris, Serverless@AWS

JJJollyjim 6y ago

I just checked cloudwatch for my rust-based function, I'm now being billed 1ms :D