5 ms·

Cool article, but shouldn’t Amazon be providing this kind of info? Surely they have this data internally.

VWWHFSfQ 5y ago
They don't want to provide it themselves because then they have to admit that the performance is abysmal. Instead they let random blogos provide this data so they can just sit back and say "you're doing it wrong."
wongarsu 5y ago
It makes Lambda look like a product with a much narrower niche than what AWS wants to sell it as. For many people knowing beforehand that cold start times are > 500ms with 256MB (quite extravagant for serving a single web request) would disqualify Lambda for any customer-serving endpoint. As it stands many get tricked into that choice if they don't perform these tests themselves.
- d0gsg0w00f 5y ago
  In my experience, if you can't say "get over it" to your customers when they complain about performance then Lambda is not the right tool. Just use EC2.
  - wongarsu 5y ago
    It's an excellent product for glue code between the various AWS services. Just about every AWS product can trigger Lambda functions, so if you want to run image recognition whenever a new image is uploaded to S3, Lambda is the way to do that. They also make great cron jobs. But for some reason Amazon likes to sell it as a way to run any web application backend, as if that was a good use case.
    - sudhirj 5y ago
      It can be. We run dynamic image resizing (have a couple million image high quality originals in S3, and customers request sizes based on their screen). Each request is handled by a lambda, and even though these are memory intensive operations we never need to worry about servers or running out or RAM or circuit breakers or anything. Whatever the load it just works. The actual operations take on the order of 100ms, so the cold start is negligible to us. And the end product is cached on a CDN anyway. Costs less than one m5.large, but at peak loads it does work 100 times what's possible on the m5.large. Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance. A single servers could resize images in sequence all day and might be cheaper than running a lambda repeatedly all day, but that's not the requirement. The requirement is to suddenly do 100 things in parallel within 100ms just once when you open the app.
      - iudqnolq 5y ago
        > Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance You're probably just simplifying, but to clarify servers can totally do multiple things at once. That's how Amazon runs multiple lambdas on one physical server.
        sudhirj 5y ago
        They can, and these are already multi core operations. If takes 4 cores 100ms to do this operation, so on a server with 4 cores doing a 100 of them takes 10 seconds, while on lambda it takes only 0.1 seconds to to them all in parallel.
unraveller 5y ago
They have never cared about this cold-start metric or the devs who do. The hope is that the first users have a degraded experience helps the next 1,000 users that minute have a perfect experience. To AWS it's like complaining about the end bit of crust in an endless loaf of sliced white bread that was baked in under 2 seconds.
- ctvo 5y ago
  This incorrect. They've made optimizations in this space, but it's a hard problem with a lot of variables. Examples include mostly solving for VPC + elastic interface provisioning, which used to take much longer and made Lambdas within VPCs unusable for customer facing APIs. The size of the individual Lambda matters quite a bit. It has to be downloaded, processed, and initialized. Latency then varies by language. They can optimize things like their own Lambda runtime that executes that code on a per language basis being quicker, but the rest are hard problems and/or requires educating customers. Their biggest problem is they oversold Lambda and serverless in my opinion, and now walk it back very slowly, buried deep in their documentation.
ignoramous 5y ago
afaik AWS doesn't publish benchmarks on runtimes; but if they did, I am sure it'd result in a lot of finger-pointing and wasted energy if they were not to normalize the process first (something like acidtests.org). That said, they do publish plenty of guidance. For ex, see Chapter 9: Optimizing serverless application performance of their well-architected series: https://aws.amazon.com/ru/blogs/compute/building-well-architected-serverless-applications-introduction/ https://aws.amazon.com/ru/blogs/compute/building-well-archit...
pid-1 5y ago
A common AWS pattern: 1 - Hey, $SERVICE looks exactly what I need. Neat! 2 - Wait, how do I do $THING with $SERVICE. No way $SERVICE can't do $THING. 3 - Realize $SERVICE is extremely limited and full of weird edge cases, get pissed. In general their docs are not transparent about the limitations of their products compared to similar non managed solutions. I sort of gave up using AWS managed services after a few years DevOpsing, except for the more flexible / battle tested ones: VPC, EC2, etc...

They don't want to provide it themselves because then they have to admit that the performance is abysmal. Instead they let random blogos provide this data so they can just sit back and say "you're doing it wrong."

wongarsu 5y ago

It makes Lambda look like a product with a much narrower niche than what AWS wants to sell it as. For many people knowing beforehand that cold start times are > 500ms with 256MB (quite extravagant for serving a single web request) would disqualify Lambda for any customer-serving endpoint. As it stands many get tricked into that choice if they don't perform these tests themselves.

d0gsg0w00f 5y ago
In my experience, if you can't say "get over it" to your customers when they complain about performance then Lambda is not the right tool. Just use EC2.
- wongarsu 5y ago
  It's an excellent product for glue code between the various AWS services. Just about every AWS product can trigger Lambda functions, so if you want to run image recognition whenever a new image is uploaded to S3, Lambda is the way to do that. They also make great cron jobs. But for some reason Amazon likes to sell it as a way to run any web application backend, as if that was a good use case.
  - sudhirj 5y ago
    It can be. We run dynamic image resizing (have a couple million image high quality originals in S3, and customers request sizes based on their screen). Each request is handled by a lambda, and even though these are memory intensive operations we never need to worry about servers or running out or RAM or circuit breakers or anything. Whatever the load it just works. The actual operations take on the order of 100ms, so the cold start is negligible to us. And the end product is cached on a CDN anyway. Costs less than one m5.large, but at peak loads it does work 100 times what's possible on the m5.large. Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance. A single servers could resize images in sequence all day and might be cheaper than running a lambda repeatedly all day, but that's not the requirement. The requirement is to suddenly do 100 things in parallel within 100ms just once when you open the app.
    - iudqnolq 5y ago
      > Say you open a page with a 100 images on it for example. With lambda the all images are resized for you in parallel, so total 100ms. If this was servers, would have to run 100 servers to give you the same performance You're probably just simplifying, but to clarify servers can totally do multiple things at once. That's how Amazon runs multiple lambdas on one physical server.
      - sudhirj 5y ago
        They can, and these are already multi core operations. If takes 4 cores 100ms to do this operation, so on a server with 4 cores doing a 100 of them takes 10 seconds, while on lambda it takes only 0.1 seconds to to them all in parallel.

unraveller 5y ago

They have never cared about this cold-start metric or the devs who do. The hope is that the first users have a degraded experience helps the next 1,000 users that minute have a perfect experience. To AWS it's like complaining about the end bit of crust in an endless loaf of sliced white bread that was baked in under 2 seconds.

ctvo 5y ago
This incorrect. They've made optimizations in this space, but it's a hard problem with a lot of variables. Examples include mostly solving for VPC + elastic interface provisioning, which used to take much longer and made Lambdas within VPCs unusable for customer facing APIs. The size of the individual Lambda matters quite a bit. It has to be downloaded, processed, and initialized. Latency then varies by language. They can optimize things like their own Lambda runtime that executes that code on a per language basis being quicker, but the rest are hard problems and/or requires educating customers. Their biggest problem is they oversold Lambda and serverless in my opinion, and now walk it back very slowly, buried deep in their documentation.

ignoramous 5y ago

afaik AWS doesn't publish benchmarks on runtimes; but if they did, I am sure it'd result in a lot of finger-pointing and wasted energy if they were not to normalize the process first (something like acidtests.org). That said, they do publish plenty of guidance. For ex, see Chapter 9: Optimizing serverless application performance of their well-architected series: https://aws.amazon.com/ru/blogs/compute/building-well-architected-serverless-applications-introduction/ https://aws.amazon.com/ru/blogs/compute/building-well-archit...

pid-1 5y ago

A common AWS pattern: 1 - Hey, $SERVICE looks exactly what I need. Neat! 2 - Wait, how do I do $THING with $SERVICE. No way $SERVICE can't do $THING. 3 - Realize $SERVICE is extremely limited and full of weird edge cases, get pissed. In general their docs are not transparent about the limitations of their products compared to similar non managed solutions. I sort of gave up using AWS managed services after a few years DevOpsing, except for the more flexible / battle tested ones: VPC, EC2, etc...