4 ms·

Sort of, but lambda-like runtimes typically act like something in between process-per-request and a long-lived persistent process. AWS lambda at least will reu

Sort of, but lambda-like runtimes typically act like something in between process-per-request and a long-lived persistent process. AWS lambda at least will reuse the same execution environment for multiple requests. So simple things like caching in a hashmap or using a database connection pool will work and avoid these startup costs. You still get CGI-like behaviour on cold starts but once warm they perform much more like a persistent process. This ends up being a pretty good trade off for many use cases because you use little resources when idle and get pretty good latency and resource usage when busy.

tsimionescu 3y ago
From my experience with a slow start AWS lambda, the cold start times really bite you every time you have a spike in concurrent requests, even a small one. We had a pretty pathological case (a Python lambda with rather a lot of code which also required a network adapter to be deployed inside a VPC), but the point still stands even for better behaved lambdas: if you have a steady 1000 concurrent connections say, you'll be OK for days. But if then 100 more concurrent connections need spinning up, you'll have 100 users taking the slow path, which may be their first experience of your service.
- kevincox 3y ago
  Yes, it is a tradeoff. I don't know about AWS Lambda specifically but this sort of behaviour can be mitigated with pre-warming. So you always maintain N% extra capacity. Then you allow the instance to start up before actually sending any requests there. Of course if you scale up >N% faster than your instances can start up you will still hit a cold start, but if your app starts relatively quickly and you maintain a decent buffer that should be fairly rare.
- Buttons840 3y ago
  I wonder what percentage of users would place the blame on your service? I know my entire life I've had unreliable Internet connections and so my instinct is to first shake my fist at my ISP.

tsimionescu 3y ago

From my experience with a slow start AWS lambda, the cold start times really bite you every time you have a spike in concurrent requests, even a small one. We had a pretty pathological case (a Python lambda with rather a lot of code which also required a network adapter to be deployed inside a VPC), but the point still stands even for better behaved lambdas: if you have a steady 1000 concurrent connections say, you'll be OK for days. But if then 100 more concurrent connections need spinning up, you'll have 100 users taking the slow path, which may be their first experience of your service.

kevincox 3y ago
Yes, it is a tradeoff. I don't know about AWS Lambda specifically but this sort of behaviour can be mitigated with pre-warming. So you always maintain N% extra capacity. Then you allow the instance to start up before actually sending any requests there. Of course if you scale up >N% faster than your instances can start up you will still hit a cold start, but if your app starts relatively quickly and you maintain a decent buffer that should be fairly rare.
Buttons840 3y ago
I wonder what percentage of users would place the blame on your service? I know my entire life I've had unreliable Internet connections and so my instinct is to first shake my fist at my ISP.