8 ms·

At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint f

At my last job we built an entire API on top of serverless. One of the things we had to figure out was this cold start time. If a user were to hit an endpoint for the first time, it would take 2x as long as it normally would at first. To combat this we wrote a "runWarm" function that kept the API alive at all times. Sure kind of defeats the purpose of serverless but hey, enterprise software.

devonkim 5y ago
To be fair, all you’d need to accomplish that without more than necessary parts for production is to ensure that the code path invoking the function is accessed via an external monitoring probe with an adjustment to SLA or SLO to account for the cold start time. Obviously not going to work for many systems, but it’s easy to forget all the side effects of the observability plane when writing applications.
deleted 5y ago
[deleted]
nodejs_rulez_1 5y ago
> ...Sure kind of defeats the purpose of serverless but hey, enterprise software. No, no, no. It's "hey, Amazon/Microsoft cloud engineering". They should be amazing with whiteboard interview exercises though.
jpdb 5y ago
> To combat this we wrote a "runWarm" function that kept the API alive at all times. This doesn't really work like you'd expect and isn't recommended, as it only helps a particular use-case. The reason is that AWS Lambda will only keep a single instance of your function alive. That means if two requests come in at the same time, you'd see a cold start on one of those invocations. Instead, you want to look at something like provisioned concurrency.
- VWWHFSfQ 5y ago
  Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely better. I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.
  - sneak 5y ago
    ...or a big latency budget? Slow start is fine for a sudden burst for a lot of use cases.
  - berkes 5y ago
    > I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic. This has been the outcome for me on several projects too. Just use loadbalanced EC2 (or EB, for simplification) and pay for a few instances running 24/7. It's actually cheaper than having a busy lambda in all my cases. The only other case (other than occasional backoffice jobs) would be long-tail stuff: an API endpoint that is used in rare situations: for example the "POST /datatakeout" or "DELETE /subscription/1337" or such. Things that might be heavy, require offbeat tools and so on. We've had them for building PDFs and .docx from reports; a feature used by <2% of the users, yet requiring all sorts of tools, from latex to pandoc.
    - tyingq 5y ago
      I would guess this situation is maintained on purpose by AWS as the upsell reason for Fargate.
    - VWWHFSfQ 5y ago
      Yeah the caveats, gotchas, and workarounds you have to do to get something reasonable running on Lambda are just goofy. At some point we just stopped and wondered why we were punishing ourselves with this stuff. We switched to a traditional webserver on regular EC2 instances and haven't looked back.
- x86_64Ubuntu 5y ago
  Here is a little AWS doc describing what parent is talking about. Personally, I had confused "provisioned concurrency" with "concurrency limit" since I don't work with cloud stuff outside of hobbying. https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/ https://aws.amazon.com/blogs/aws/new-provisioned-concurrency...
- raro11 5y ago
  Have your runWarm sleep for 500ms and execute 50 of them concurrently. As long as none of the functions are finished and you start a new one you get a new instance, at least that's what I think. You can get 50 hot instances that way no? I'd rather scale per connections. Have a lambda instance do 50 concurrent requests. Something like https://fly.io https://fly.io but cheaper.
bob1029 5y ago
Is there any guarantee of maximum startup time, or is there some upper timeout bound that you always have to anticipate in the worst case?
- tyingq 5y ago
  Not that I ever saw. They have made many improvements. But a cold start time of 2 minutes wasn't considered an bug or issue before they fixed the VPC/Lambda interconnect.
kylecordes 5y ago
Same thing here, on a different cloud. When we notice someone is using a form, we fire a no-op request to the function that will handle the data from the form so that it is less likely to be cold when the user is ready to proceed. (We could get better results by switching to a different implementation language; but we have a body of code already working correctly aside from the extra second or two of cold start.)
Unklejoe 5y ago
That reminds me of a custom Linux device driver that I worked with in the past. It implemented "mmap" so that a user application could map a ring buffer into userspace for zero-copy transfers. It used lazy mapping in the sense that it relied on the page fault handler to get triggered to map each page in as they were touched. This resulted in a latency increase for the very first accesses, but then it was fast after that since the pages stayed mapped in. The solution? Read the entire ring buffer one time during startup to force all pages to get mapped in. I eventually changed the driver to just map them all in at once.
k__ 5y ago
If you build an API and are concerned about cold-starts you could look at Lambda@Edge and "CloudFront Functions". I could imagine these to perform better. If you aren't married to AWS, then Cloudflare Workers could also be worth a shot.
- dmw_ng 5y ago
  Lambda@Edge helps with latency, definitely not with cold start times. You also can't buy provisioned Lambda@Edge, so for low traffic scenarios it's even worse than typical Lambda (where you can easily provision capacity, or keep on-demand capacity warm, which is not so cheap or easy when that must be done across every CloudFront cache region). For a low traffic environment, running e.g. 3-5 regular provisioned Lambda functions in different regions will produce a much more sensible latency distribution for end users than Edge would. CloudFront Functions have no cold start, but their execution time is sorely restricted (1ms IIRC). You can't do much with them except origin selection, header tweaks or generating redirects, and there is no network or filesystem IO whatsoever.
nuerow 5y ago
> To combat this we wrote a "runWarm" function that kept the API alive at all times. This sort of hack is not needed in AWS Lambdas, as they support provisioned concurrency.
- davidjfelix 5y ago
  Nor does it actually work. If you have a synthetic "runWarm" event, you'll trigger one concurrent lambda to stay warm. This helps if your cold start time is long and your average invoke time is short but you're just levying the cold start tax to the second concurrent user. There's no reasonable way to keep a concurrency > 1 warm with synthetic events without negatively impacting your cold start percentage for users. Provisioned concurrency is the correct solution and I'll remind everyone here that you can put provisioned concurrency in an autoscaling group, since the comments here seem to be saying keeping 100 lambdas warm is worse than a server that can handle 100 concurrent users (DUH!)
ctvo 5y ago
What you and others are doing is attempting to predict your peak traffic when you take this approach. It may work for some companies, but more commonly in my experience, it hides P99+ tail latency from companies that may not instrument deeply (and they think the problem is solved). The rate at which you execute `runWarm` is the peak traffic you're expecting. A request comes in over that threshold and you'll still experience cold start latency. Provisioned concurrency doesn't change this, but it does move the complexity of `runWarm` to the Lambda team and gives you more control (give me a pool of 50 warmed Lambdas vs. me trying too run `runWarm` enough to keep 50 warmed myself). That's valuable in a lot of use cases. At the end of the day you're still in the game of predicting peak traffic and paying (a lot) for it. We're almost always trying to predict peak traffic though! The difference is using a course grain computing platform, like EC2 for example, where a single box can handle hundreds++ requests per second, gives you more room for error, and is cheaper. There are a lot of other trade-offs to consider. My biggest issue is this isn't enumerated clearly by AWS, and I run into way too many people who have footgun themselves unnecessarily with Lambda.
mikesabbagh 5y ago
> To combat this we wrote a "runWarm" function that kept the API alive at all times. maybe one should create 1 huge lambda instead of several hundred
- bmm6o 5y ago
  Cold starts aren't just going from 0 to 1, you hit it anytime you scale. Putting everything in a single lambda likely means more scaling up.
lostcolony 5y ago
"To combat this" Did you actually need to? That's one of the things that always threw me with complaints about cold starts - how many apps/etc do I use daily where I interact, and there's a 10 second delay before something happens? The answer: quite a lot. Yeah, we can do better. And in fact, with Serverless, -most users will experience better-. It's only when load is increasing that you see those delays, and then it's still only a delay. Not shed load. The fact I can experience that delay easily in dev makes people think it's going to be a huge problem, but, A. In real use it probably isn't as common as it is in dev (since you have way more traffic) B. You can design to minimize it (different API endpoints can hit the same lambda and be routed to the right handler there, making it more likely to be hot), C. It forces you to test and plan for worst case from the beginning (rather than at the end where you've built something, and now have to load test it). Not to say to use it all the time, of course; there are plenty of scenarios where the cost, the delay, etc, are non-starters. But there are also plenty of scenarios where an engineer's instinctual reaction would be "too slow", but in reality it's fine; your p95 is going to look great, and only your P99 is going to look bad (on that note, a slow API response accompanied with a spinner is a very different thing from a UX perspective than a slow page load with no indication of progress), and even then it's predictable when it happens, and it forces function scale out rather than tanking a service. Of course, it's often not obvious upfront which scenarios those would be until/unless you try it, and that's definitely a barrier.
- deleted 5y ago
  [deleted]
- jacurtis 5y ago
  There is actually a really awesome middle-ground that AWS offers that no one seems to talk about. That is using ECS + Fargate. This gives you (IMHO) the best of both worlds between Lambda and EC2. ECS is Elastic Container Service. Think docker/podman containers. You can even pull from Dockerhub or ECR (Elastic Container Registry - amazon's version of dockerhub). ECS can then deploy to either a traditional EC2 compute instance (giving you a standard containerization deployment) or to "Fargate". Fargate is a serverless container compute instance. It is like serverless EC2. You get the "serverless" benefits of Lambda, but it is always-on. It has automatic scaling, so it can scale up and down with traffic (all of which is configured in ECS). You don't need to manage security updates of the underlying compute instance or manage the system. You get high-availability and fault tolerance "for free". But at the end of the day, its basically a non-managed EC2 instance. You can choose the ram/cpu options that you need for your Fargate just like any other compute instance. My recommendation is go as small as possible and rely on horizontal scaling instead of vertical. This keeps costs as low as possible. When I hear people trying to keep Lambdas running indefinitely, it really defeats the purpose of Lambda. Lambda has plenty of benefits, but it is best used for functions that run intermittently and are isolated. If you want the serverless benefits of Lambda, but want to have the benefits of a traditional server too, then you need to look at Fargate. And of course there is a world where you combine the two. Maybe you have an authentication service that needs to run 24/7. Run it via ECS+Fargate. Maybe your primary API should also run on Fargate. But then when you need to boot up a bunch of batch processing at midnight each night to send out invoices, those can use Lambdas. They do their job and then go to sleep until the next day. I should also add that the developer experience is far superior going the ECS+Fargate route over Lambda. I have built extensive APIs in Lambda and they are so difficult to debug and you always feel like you are coding with one hand tied behind your back. But with ECS+Fargate you just build projects as you normally would, with your traditional environment. You can do live testing locally just like any other container project. Run docker or podman on your system using an Amazon Linux, Alpine Linux, CentOS base. And that same environment will match your Fargate deployment. It makes the developer experience much better.

devonkim 5y ago

To be fair, all you’d need to accomplish that without more than necessary parts for production is to ensure that the code path invoking the function is accessed via an external monitoring probe with an adjustment to SLA or SLO to account for the cold start time. Obviously not going to work for many systems, but it’s easy to forget all the side effects of the observability plane when writing applications.

deleted 5y ago

[deleted]

nodejs_rulez_1 5y ago

> ...Sure kind of defeats the purpose of serverless but hey, enterprise software. No, no, no. It's "hey, Amazon/Microsoft cloud engineering". They should be amazing with whiteboard interview exercises though.

jpdb 5y ago

> To combat this we wrote a "runWarm" function that kept the API alive at all times. This doesn't really work like you'd expect and isn't recommended, as it only helps a particular use-case. The reason is that AWS Lambda will only keep a single instance of your function alive. That means if two requests come in at the same time, you'd see a cold start on one of those invocations. Instead, you want to look at something like provisioned concurrency.

VWWHFSfQ 5y ago
Provisioned concurrency is insanely expensive. If you have any kind of a thundering herd access pattern then Lambda is a complete non-starter because of the warm-up and scaling characteristics. We eventually just put an nginx/openresty server on a regular medium EC2 instance and got rid of Lambda from our stack completely and now we're paying about 1/300th the cost we were previously and the performance is infinitely better. I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic.
- sneak 5y ago
  ...or a big latency budget? Slow start is fine for a sudden burst for a lot of use cases.
- berkes 5y ago
  > I'm sure it has some use-cases in some kind of backoffice task queue scenario, but Lambda is nearly unusable in a web context unless you have a very trivial amount of traffic. This has been the outcome for me on several projects too. Just use loadbalanced EC2 (or EB, for simplification) and pay for a few instances running 24/7. It's actually cheaper than having a busy lambda in all my cases. The only other case (other than occasional backoffice jobs) would be long-tail stuff: an API endpoint that is used in rare situations: for example the "POST /datatakeout" or "DELETE /subscription/1337" or such. Things that might be heavy, require offbeat tools and so on. We've had them for building PDFs and .docx from reports; a feature used by <2% of the users, yet requiring all sorts of tools, from latex to pandoc.
  - tyingq 5y ago
    I would guess this situation is maintained on purpose by AWS as the upsell reason for Fargate.
  - VWWHFSfQ 5y ago
    Yeah the caveats, gotchas, and workarounds you have to do to get something reasonable running on Lambda are just goofy. At some point we just stopped and wondered why we were punishing ourselves with this stuff. We switched to a traditional webserver on regular EC2 instances and haven't looked back.
x86_64Ubuntu 5y ago
Here is a little AWS doc describing what parent is talking about. Personally, I had confused "provisioned concurrency" with "concurrency limit" since I don't work with cloud stuff outside of hobbying. https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/ https://aws.amazon.com/blogs/aws/new-provisioned-concurrency...
raro11 5y ago
Have your runWarm sleep for 500ms and execute 50 of them concurrently. As long as none of the functions are finished and you start a new one you get a new instance, at least that's what I think. You can get 50 hot instances that way no? I'd rather scale per connections. Have a lambda instance do 50 concurrent requests. Something like https://fly.io https://fly.io but cheaper.

bob1029 5y ago

Is there any guarantee of maximum startup time, or is there some upper timeout bound that you always have to anticipate in the worst case?

tyingq 5y ago
Not that I ever saw. They have made many improvements. But a cold start time of 2 minutes wasn't considered an bug or issue before they fixed the VPC/Lambda interconnect.

kylecordes 5y ago

Same thing here, on a different cloud. When we notice someone is using a form, we fire a no-op request to the function that will handle the data from the form so that it is less likely to be cold when the user is ready to proceed. (We could get better results by switching to a different implementation language; but we have a body of code already working correctly aside from the extra second or two of cold start.)

Unklejoe 5y ago

That reminds me of a custom Linux device driver that I worked with in the past. It implemented "mmap" so that a user application could map a ring buffer into userspace for zero-copy transfers. It used lazy mapping in the sense that it relied on the page fault handler to get triggered to map each page in as they were touched. This resulted in a latency increase for the very first accesses, but then it was fast after that since the pages stayed mapped in. The solution? Read the entire ring buffer one time during startup to force all pages to get mapped in. I eventually changed the driver to just map them all in at once.

k__ 5y ago

If you build an API and are concerned about cold-starts you could look at Lambda@Edge and "CloudFront Functions". I could imagine these to perform better. If you aren't married to AWS, then Cloudflare Workers could also be worth a shot.

dmw_ng 5y ago
Lambda@Edge helps with latency, definitely not with cold start times. You also can't buy provisioned Lambda@Edge, so for low traffic scenarios it's even worse than typical Lambda (where you can easily provision capacity, or keep on-demand capacity warm, which is not so cheap or easy when that must be done across every CloudFront cache region). For a low traffic environment, running e.g. 3-5 regular provisioned Lambda functions in different regions will produce a much more sensible latency distribution for end users than Edge would. CloudFront Functions have no cold start, but their execution time is sorely restricted (1ms IIRC). You can't do much with them except origin selection, header tweaks or generating redirects, and there is no network or filesystem IO whatsoever.

nuerow 5y ago

> To combat this we wrote a "runWarm" function that kept the API alive at all times. This sort of hack is not needed in AWS Lambdas, as they support provisioned concurrency.

davidjfelix 5y ago
Nor does it actually work. If you have a synthetic "runWarm" event, you'll trigger one concurrent lambda to stay warm. This helps if your cold start time is long and your average invoke time is short but you're just levying the cold start tax to the second concurrent user. There's no reasonable way to keep a concurrency > 1 warm with synthetic events without negatively impacting your cold start percentage for users. Provisioned concurrency is the correct solution and I'll remind everyone here that you can put provisioned concurrency in an autoscaling group, since the comments here seem to be saying keeping 100 lambdas warm is worse than a server that can handle 100 concurrent users (DUH!)

ctvo 5y ago

What you and others are doing is attempting to predict your peak traffic when you take this approach. It may work for some companies, but more commonly in my experience, it hides P99+ tail latency from companies that may not instrument deeply (and they think the problem is solved). The rate at which you execute `runWarm` is the peak traffic you're expecting. A request comes in over that threshold and you'll still experience cold start latency. Provisioned concurrency doesn't change this, but it does move the complexity of `runWarm` to the Lambda team and gives you more control (give me a pool of 50 warmed Lambdas vs. me trying too run `runWarm` enough to keep 50 warmed myself). That's valuable in a lot of use cases. At the end of the day you're still in the game of predicting peak traffic and paying (a lot) for it. We're almost always trying to predict peak traffic though! The difference is using a course grain computing platform, like EC2 for example, where a single box can handle hundreds++ requests per second, gives you more room for error, and is cheaper. There are a lot of other trade-offs to consider. My biggest issue is this isn't enumerated clearly by AWS, and I run into way too many people who have footgun themselves unnecessarily with Lambda.

mikesabbagh 5y ago

> To combat this we wrote a "runWarm" function that kept the API alive at all times. maybe one should create 1 huge lambda instead of several hundred

bmm6o 5y ago
Cold starts aren't just going from 0 to 1, you hit it anytime you scale. Putting everything in a single lambda likely means more scaling up.

lostcolony 5y ago

"To combat this" Did you actually need to? That's one of the things that always threw me with complaints about cold starts - how many apps/etc do I use daily where I interact, and there's a 10 second delay before something happens? The answer: quite a lot. Yeah, we can do better. And in fact, with Serverless, -most users will experience better-. It's only when load is increasing that you see those delays, and then it's still only a delay. Not shed load. The fact I can experience that delay easily in dev makes people think it's going to be a huge problem, but, A. In real use it probably isn't as common as it is in dev (since you have way more traffic) B. You can design to minimize it (different API endpoints can hit the same lambda and be routed to the right handler there, making it more likely to be hot), C. It forces you to test and plan for worst case from the beginning (rather than at the end where you've built something, and now have to load test it). Not to say to use it all the time, of course; there are plenty of scenarios where the cost, the delay, etc, are non-starters. But there are also plenty of scenarios where an engineer's instinctual reaction would be "too slow", but in reality it's fine; your p95 is going to look great, and only your P99 is going to look bad (on that note, a slow API response accompanied with a spinner is a very different thing from a UX perspective than a slow page load with no indication of progress), and even then it's predictable when it happens, and it forces function scale out rather than tanking a service. Of course, it's often not obvious upfront which scenarios those would be until/unless you try it, and that's definitely a barrier.

deleted 5y ago
[deleted]
jacurtis 5y ago
There is actually a really awesome middle-ground that AWS offers that no one seems to talk about. That is using ECS + Fargate. This gives you (IMHO) the best of both worlds between Lambda and EC2. ECS is Elastic Container Service. Think docker/podman containers. You can even pull from Dockerhub or ECR (Elastic Container Registry - amazon's version of dockerhub). ECS can then deploy to either a traditional EC2 compute instance (giving you a standard containerization deployment) or to "Fargate". Fargate is a serverless container compute instance. It is like serverless EC2. You get the "serverless" benefits of Lambda, but it is always-on. It has automatic scaling, so it can scale up and down with traffic (all of which is configured in ECS). You don't need to manage security updates of the underlying compute instance or manage the system. You get high-availability and fault tolerance "for free". But at the end of the day, its basically a non-managed EC2 instance. You can choose the ram/cpu options that you need for your Fargate just like any other compute instance. My recommendation is go as small as possible and rely on horizontal scaling instead of vertical. This keeps costs as low as possible. When I hear people trying to keep Lambdas running indefinitely, it really defeats the purpose of Lambda. Lambda has plenty of benefits, but it is best used for functions that run intermittently and are isolated. If you want the serverless benefits of Lambda, but want to have the benefits of a traditional server too, then you need to look at Fargate. And of course there is a world where you combine the two. Maybe you have an authentication service that needs to run 24/7. Run it via ECS+Fargate. Maybe your primary API should also run on Fargate. But then when you need to boot up a bunch of batch processing at midnight each night to send out invoices, those can use Lambdas. They do their job and then go to sleep until the next day. I should also add that the developer experience is far superior going the ECS+Fargate route over Lambda. I have built extensive APIs in Lambda and they are so difficult to debug and you always feel like you are coding with one hand tied behind your back. But with ECS+Fargate you just build projects as you normally would, with your traditional environment. You can do live testing locally just like any other container project. Run docker or podman on your system using an Amazon Linux, Alpine Linux, CentOS base. And that same environment will match your Fargate deployment. It makes the developer experience much better.