4 ms·

Comparing AWS Lambda performance of Node.js, Python, Java, C# and Go

113 points by dfirment 9y ago

QuinnyPig 9y ago
Go on par with Java? .NET 2.0 beating everything else out?! I feel like my world has just flipped upside down.
sjellis 9y ago
To the extent that this benchmark is valid, this is interesting: "In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js. Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages." TLDR: You may be able to use a "slower" language for Lambda without necessarily losing much.
- singularity2001 9y ago
  >> You may be able to use a "slower" language for Lambda at least for small projects
petethepig 9y ago
This is a good example of an absolutely pointless comparison. The author is making some bold statements: > Go performance is comparable to Java > .Net Core 2.0 can execute up to 3x faster than Go. > Consistent performance of compiled vs dynamic But what are they measuring? How fast can a lambda function return "200 OK hello"? This is just sad.
- hugelgupf 9y ago
  They are effectively measuring how fast AWS infrastructure can reach whatever it needs (VM?) for each language. But these are not even cold start times. I'm assuming after the first request this is using the same VM (/backend job, whatever) to reply, so now it just depends on latency to get to that VM. (I'm assuming the runtimes' ability to return "200 OK" is similar and the latency of that is negligible compared to AWS infra that is involved.) Not completely meaningless, but not super useful either, esp without any details while making those bold statements. This is not "Go is equal to Java". This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java". Article is unclear on that. Maybe this tells us AWS lambda backend infra is way different for .NET Core than Go and Java? ¯\_(ツ)_/¯
  - petethepig 9y ago
    > This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java" I wouldn't even go that far. They are not measuring anything that any real-world application would do. Heck, it's not even a todo app, it's just a "200 OK hello". It's more like "AWS lambda function invocation overhead for Go is comparable to AWS lambda function invocation overhead for Java".
    - weego 9y ago
      Sure but if you start adding in language specific Web frameworks / json handling / db work you're no longer testing AWS. I agree there's questionable value here but at least it's bring constrained to as few variables as possible.
      - lsaferite 9y ago
        Benchmarking the lambda-add latency is something that is useful as long as it's called out as such a benchmark.
- IloveHN84 9y ago
  It's a pity there's no C/C++ example as well (maybe with openssl?) For low level comparison
deleted 9y ago
[deleted]
thepumpkin1979 9y ago
Ok, now being serious, they need to include a real example here. Whenever I run a Lambda function is to process something and make a call to at least one external service. I was a .NET developer for a long time and I seriously doubt .NET Core has been improved to the point where blocking http calls are handled better than in Go runtime. I may believe .NET Core may have some performance advantages in some CPU bound scenarios, but nothing more. Until then, this benchmark proves nothing.
- jinder 9y ago
  In which scenario would you use a blocking http call?
  - avita1 9y ago
    Not necessarily HTTP, but I imagine hitting some sort of database from a lambda is not uncommon.
    - Jtsummers 9y ago
      Isn't the intent with a system like Lambda that you'd not do any kind of blocking action. Instead, you'd trigger some event that the AWS database solutions would receive. Once the DB has finished the query it sends the data back and triggers another lambda function. EDIT: I write this as someone who does no web programming and has only toyed with AWS. This was my understanding from what I read, but not something I actually made.
      - SideburnsOfDoom 9y ago
        The relevant methods in the AWS .Net SDK are all awaitable, i.e. not blocking. Even if you did block on waiting for it, an operation on DynamoDB such as writing a record should typically take under 10ms, which would be ok. So there is no reason at all not to do it in a lambda.
      - lostcolony 9y ago
        No. "Once the DB has finished the query it sends the data back and triggers another lambda function" What mechanism is the DB 'sending' data via? Is it just the response part of the initial query? Then it's executing in the context of the same lambda function. You -could- then call another lambda function from the running lambda function (but why?), but more, what was the function that executed the query doing while it waited? If it had something it -could- be doing, it should be doing it, so non-blocking is important. If it had nothing it could be doing, it literally does not matter if it was a blocking call or not; it doesn't effect anything. Now, if you meant you have a trigger in the DB that calls another lambda function, such that the original lambda could complete, yes, of course you could do that. However, if this was a GET style call, not a PUT, your options to get the data back to the user are limited if you don't just keep the REST request open and respond that way. Certainly, any other solution is more complex, probably more expensive, and may not be possible.
dmitriid 9y ago
> we are no longer observing any significant variances between Java and the newer compiled languages <proceeds to show averages instead of mean and/or 90th percentile even though maximum spikes are easily orders of magnitude higher than average>
nzoschke 9y ago
I have been doing a lot of experiments with Go and Lambda and have an open source boilerplate app here: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas This benchmark leaves some room for improvement. With Lambda and X-ray you could really deep dive into where time is spent in a request. Here’s an example trace for a Golang function that does some actual work against KMS and DynamoDB: https://twitter.com/nzoschke/status/969598630033162240?s=20 https://twitter.com/nzoschke/status/969598630033162240?s=20 One thing I found surprising about the Go support is that the ‘lambda.Start’ helper uses reflection to invoke your go func. This could contribute to some of the cold start time.
- dilap 9y ago
  Yeah that certainly seems...non ideal.
chrissnell 9y ago
Lambdas bill at a 100ms minimum. What's the point of a few ms of difference here, unless you're running some computationally-intensive code? If that's what you're doing, the efficiency of the language itself is going to be what matters, not Lambda's speed of executing the runtime.
igotsideas 9y ago
What would be the right way to test this?
tofflos 9y ago
Why the hate? The methodology is sound, the results are interesting and they show that there is some room for improvement in language support. Also the source code has been provided in case anyone wants to alter the benchmark to be more comparable to their workloads. Hopefully this sparks a race between the different teams within Amazon. Great job! Oh! These test scenarios seem to have a pretty good acceptance in case you decide to do another benchmark in the future: https://www.techempower.com/benchmarks/#section=code&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=code&hw=ph&t....
stunt 9y ago
Comparing their open-source ecosystem, I still prefer to build my lambdas with NodeJS (or even Python) over C#. (if my only concern is performance)
equalunique 9y ago
I am glad that the person who created this intended for the results to be reproducible. I will try my hand at what's in the git repo, and perhaps compare with some other languages too.

113 points by dfirment 9y ago

Go on par with Java? .NET 2.0 beating everything else out?! I feel like my world has just flipped upside down.

To the extent that this benchmark is valid, this is interesting: "In the original 2017 benchmark results, the compiled languages of Java and .Net Core 1.0 were clearly more consistent compared to the dynamic languages — Python and Node.js. Based on the latest 2018 performance testing results — we are no longer observing any significant variances between Java and the newer compiled languages (.Net Core 2.0 and Go), and the original dynamic languages." TLDR: You may be able to use a "slower" language for Lambda without necessarily losing much.

singularity2001 9y ago
>> You may be able to use a "slower" language for Lambda at least for small projects

petethepig 9y ago

This is a good example of an absolutely pointless comparison. The author is making some bold statements: > Go performance is comparable to Java > .Net Core 2.0 can execute up to 3x faster than Go. > Consistent performance of compiled vs dynamic But what are they measuring? How fast can a lambda function return "200 OK hello"? This is just sad.

hugelgupf 9y ago
They are effectively measuring how fast AWS infrastructure can reach whatever it needs (VM?) for each language. But these are not even cold start times. I'm assuming after the first request this is using the same VM (/backend job, whatever) to reply, so now it just depends on latency to get to that VM. (I'm assuming the runtimes' ability to return "200 OK" is similar and the latency of that is negligible compared to AWS infra that is involved.) Not completely meaningless, but not super useful either, esp without any details while making those bold statements. This is not "Go is equal to Java". This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java". Article is unclear on that. Maybe this tells us AWS lambda backend infra is way different for .NET Core than Go and Java? ¯\_(ツ)_/¯
- petethepig 9y ago
  > This is "AWS Lambda infrastructure for Go is comparable to AWS infra for Java" I wouldn't even go that far. They are not measuring anything that any real-world application would do. Heck, it's not even a todo app, it's just a "200 OK hello". It's more like "AWS lambda function invocation overhead for Go is comparable to AWS lambda function invocation overhead for Java".
  - weego 9y ago
    Sure but if you start adding in language specific Web frameworks / json handling / db work you're no longer testing AWS. I agree there's questionable value here but at least it's bring constrained to as few variables as possible.
    - lsaferite 9y ago
      Benchmarking the lambda-add latency is something that is useful as long as it's called out as such a benchmark.
IloveHN84 9y ago
It's a pity there's no C/C++ example as well (maybe with openssl?) For low level comparison

deleted 9y ago

[deleted]

thepumpkin1979 9y ago

Ok, now being serious, they need to include a real example here. Whenever I run a Lambda function is to process something and make a call to at least one external service. I was a .NET developer for a long time and I seriously doubt .NET Core has been improved to the point where blocking http calls are handled better than in Go runtime. I may believe .NET Core may have some performance advantages in some CPU bound scenarios, but nothing more. Until then, this benchmark proves nothing.

jinder 9y ago
In which scenario would you use a blocking http call?
- avita1 9y ago
  Not necessarily HTTP, but I imagine hitting some sort of database from a lambda is not uncommon.
  - Jtsummers 9y ago
    Isn't the intent with a system like Lambda that you'd not do any kind of blocking action. Instead, you'd trigger some event that the AWS database solutions would receive. Once the DB has finished the query it sends the data back and triggers another lambda function. EDIT: I write this as someone who does no web programming and has only toyed with AWS. This was my understanding from what I read, but not something I actually made.
    - SideburnsOfDoom 9y ago
      The relevant methods in the AWS .Net SDK are all awaitable, i.e. not blocking. Even if you did block on waiting for it, an operation on DynamoDB such as writing a record should typically take under 10ms, which would be ok. So there is no reason at all not to do it in a lambda.
    - lostcolony 9y ago
      No. "Once the DB has finished the query it sends the data back and triggers another lambda function" What mechanism is the DB 'sending' data via? Is it just the response part of the initial query? Then it's executing in the context of the same lambda function. You -could- then call another lambda function from the running lambda function (but why?), but more, what was the function that executed the query doing while it waited? If it had something it -could- be doing, it should be doing it, so non-blocking is important. If it had nothing it could be doing, it literally does not matter if it was a blocking call or not; it doesn't effect anything. Now, if you meant you have a trigger in the DB that calls another lambda function, such that the original lambda could complete, yes, of course you could do that. However, if this was a GET style call, not a PUT, your options to get the data back to the user are limited if you don't just keep the REST request open and respond that way. Certainly, any other solution is more complex, probably more expensive, and may not be possible.

dmitriid 9y ago

> we are no longer observing any significant variances between Java and the newer compiled languages <proceeds to show averages instead of mean and/or 90th percentile even though maximum spikes are easily orders of magnitude higher than average>

nzoschke 9y ago

I have been doing a lot of experiments with Go and Lambda and have an open source boilerplate app here: https://github.com/nzoschke/gofaas https://github.com/nzoschke/gofaas This benchmark leaves some room for improvement. With Lambda and X-ray you could really deep dive into where time is spent in a request. Here’s an example trace for a Golang function that does some actual work against KMS and DynamoDB: https://twitter.com/nzoschke/status/969598630033162240?s=20 https://twitter.com/nzoschke/status/969598630033162240?s=20 One thing I found surprising about the Go support is that the ‘lambda.Start’ helper uses reflection to invoke your go func. This could contribute to some of the cold start time.

dilap 9y ago
Yeah that certainly seems...non ideal.

chrissnell 9y ago

Lambdas bill at a 100ms minimum. What's the point of a few ms of difference here, unless you're running some computationally-intensive code? If that's what you're doing, the efficiency of the language itself is going to be what matters, not Lambda's speed of executing the runtime.

igotsideas 9y ago

What would be the right way to test this?

tofflos 9y ago

Why the hate? The methodology is sound, the results are interesting and they show that there is some room for improvement in language support. Also the source code has been provided in case anyone wants to alter the benchmark to be more comparable to their workloads. Hopefully this sparks a race between the different teams within Amazon. Great job! Oh! These test scenarios seem to have a pretty good acceptance in case you decide to do another benchmark in the future: https://www.techempower.com/benchmarks/#section=code&hw=ph&test=fortune https://www.techempower.com/benchmarks/#section=code&hw=ph&t....

stunt 9y ago

Comparing their open-source ecosystem, I still prefer to build my lambdas with NodeJS (or even Python) over C#. (if my only concern is performance)

equalunique 9y ago

I am glad that the person who created this intended for the results to be reproducible. I will try my hand at what's in the git repo, and perhaps compare with some other languages too.