3 ms·

Two things. 1- It's amazing that Java has the fastest cold start time! Faster than Nodejs.[1] That's exactly the opposite of what I've heard before. 2- I am s

Two things. 1- It's amazing that Java has the fastest cold start time! Faster than Nodejs.[1] That's exactly the opposite of what I've heard before. 2- I am so tired of hearing about cold start times for dormant apps as if that is the only cold start scenario. It is arguably a worse problem to have cold starts when scaling! What do I mean by cold starts when scaling? You adopt serverless. Things go great. Your app is never dormant. You're not serverless because you want to shave costs for infrequently unused apps. You're serverless so you have infinite scale and pay by the millisecond and minimal devops and so on. But whenever you have a burst of scale and Lambda needs to spin up more instances... some of your unlucky users are going to hit that cold start. And this hack of keeping an instance warm would do nothing to solve that. I mean, do they? Do we know? It's possible that AWS warms up instances before throwing traffic at them but.. has anyone looked at this? [1] https://mikhail.io/2018/08/serverless-cold-start-war//coldstart-dependencies.png https://mikhail.io/2018/08/serverless-cold-start-war//coldst...

TrueTeller 8y ago
Yes, one could prewarm N instances by sending N parallel requests every 5 minutes. You have to know N though :) For most apps, scale-out cold start won't be too much of a deal breaker: longer instance lifetime + shorter start will do the trick. And wait for the next wave of optimizations from vendors, I'm sure there's more to come. P.S. I'm the author of OP, thanks for reading!
- abalone 8y ago
  > Yes, one could prewarm N instances by sending N parallel requests every 5 minutes. Uhh.. you’re just speculating here right? I seriously doubt Lambda spins up a whole new container for every single “parallel” request in a short burst. There’s probably a little bit of queuing and/or they aren’t exactly parallel. I think it’s something the vendors need to solve, like by directly prewarming prior to throwing traffic at it. Oh and thank you for the detailed post!
  - mikestaub 8y ago
    I agree the vendors needs to solve this. In the meantime I think you can hack it by using Kinesis shards ( one lambda per shard ) if you know you will need many lambdas in advance. https://github.com/awslabs/aws-lambda-kinesis-prewarming https://github.com/awslabs/aws-lambda-kinesis-prewarming
jacques_chester 8y ago
> But whenever you have a burst of scale and Lambda needs to spin up more instances... some of your unlucky users are going to hit that cold start. And this hack of keeping an instance warm would do nothing to solve that. I've been doing a bunch of reading as a side-effect of being in and around the riff and Knative autoscaler efforts. What you're describing is known in other professions as a "stock-out". The good news is: there are existing models for answering this kind of question. From what I've seen the "order up to" model is a fit, but I've yet to find time to work on testing that theory. The bad news is: this problem never goes away. You are always going to be oversupplied or undersupplied. Autoscalers don't let you break Little's Law or overturn causality. The good or bad news, depending on how you think: this tradeoff can be tuned. You can choose an acceptable probability of running out of running instances vs the acceptable average level of utilisation. That tradeoff is purely economic, it is a business decision, not an engineering decision. An autoscaler cannot throw the bones, gaze into the crystal ball and mystically divine your intentions. A human will still be responsible for the decisions that matter.
akvadrako 8y ago
That chart is quite strange. Why would JS start take 500ms (on my system it's 80ms) and be slower to start with more dependencies? Maybe it's because the bulk of the time is just copying the deployment artifact to the local disk. In that case the overriding factor is the size of the package.
- icebraining 8y ago
  If you require() the libraries at the top, as it seems to be common in Node applications, it makes sense that more dependencies add to the cold start time.
village-idiot 8y ago
A shockingly large number of people run around repeating what they remember from Java 6 or earlier, without bothering to realize tha Java and the JVM have changed a lot.
icebraining 8y ago
I don't know how serverless platforms deal with it, but startup times could be "solved" by starting up the function and cloning it when it's ready to process requests, as Unix daemons have done for a long time.

TrueTeller 8y ago

Yes, one could prewarm N instances by sending N parallel requests every 5 minutes. You have to know N though :) For most apps, scale-out cold start won't be too much of a deal breaker: longer instance lifetime + shorter start will do the trick. And wait for the next wave of optimizations from vendors, I'm sure there's more to come. P.S. I'm the author of OP, thanks for reading!

abalone 8y ago
> Yes, one could prewarm N instances by sending N parallel requests every 5 minutes. Uhh.. you’re just speculating here right? I seriously doubt Lambda spins up a whole new container for every single “parallel” request in a short burst. There’s probably a little bit of queuing and/or they aren’t exactly parallel. I think it’s something the vendors need to solve, like by directly prewarming prior to throwing traffic at it. Oh and thank you for the detailed post!
- mikestaub 8y ago
  I agree the vendors needs to solve this. In the meantime I think you can hack it by using Kinesis shards ( one lambda per shard ) if you know you will need many lambdas in advance. https://github.com/awslabs/aws-lambda-kinesis-prewarming https://github.com/awslabs/aws-lambda-kinesis-prewarming

jacques_chester 8y ago

> But whenever you have a burst of scale and Lambda needs to spin up more instances... some of your unlucky users are going to hit that cold start. And this hack of keeping an instance warm would do nothing to solve that. I've been doing a bunch of reading as a side-effect of being in and around the riff and Knative autoscaler efforts. What you're describing is known in other professions as a "stock-out". The good news is: there are existing models for answering this kind of question. From what I've seen the "order up to" model is a fit, but I've yet to find time to work on testing that theory. The bad news is: this problem never goes away. You are always going to be oversupplied or undersupplied. Autoscalers don't let you break Little's Law or overturn causality. The good or bad news, depending on how you think: this tradeoff can be tuned. You can choose an acceptable probability of running out of running instances vs the acceptable average level of utilisation. That tradeoff is purely economic, it is a business decision, not an engineering decision. An autoscaler cannot throw the bones, gaze into the crystal ball and mystically divine your intentions. A human will still be responsible for the decisions that matter.

akvadrako 8y ago

That chart is quite strange. Why would JS start take 500ms (on my system it's 80ms) and be slower to start with more dependencies? Maybe it's because the bulk of the time is just copying the deployment artifact to the local disk. In that case the overriding factor is the size of the package.

icebraining 8y ago
If you require() the libraries at the top, as it seems to be common in Node applications, it makes sense that more dependencies add to the cold start time.

village-idiot 8y ago

A shockingly large number of people run around repeating what they remember from Java 6 or earlier, without bothering to realize tha Java and the JVM have changed a lot.

icebraining 8y ago

I don't know how serverless platforms deal with it, but startup times could be "solved" by starting up the function and cloning it when it's ready to process requests, as Unix daemons have done for a long time.