4 ms·

From the article: > To optimize cold start overheads, Fission keeps a running pool of containers for each environment. When a request for a function comes in,

by ekidd 10y ago

From the article: > To optimize cold start overheads, Fission keeps a running pool of containers for each environment. When a request for a function comes in, Fission doesn't have to deploy a new container -- it just chooses one that's already running, copies the function into the container, loads it dynamically, and routes the request to that instance. The overhead of this process takes on the order of 100msec for NodeJS and Python functions. 100 ms of overhead is pretty steep. Google recommends a limit of twice that for the entire server response (https://developers.google.com/speed/docs/insights/Server#recommendations https://developers.google.com/speed/docs/insights/Server#rec...): > You should reduce your server response time under 200ms Still, I've heard that AWS Lambda may run around 200 ms, so perhaps Fission is doing OK here. But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. With compiled web server frameworks, it's feasible to handle over 250,000 (very simple) requests per second, or thousands of requests while Fission is trying to figure out where to send your source code. It doesn't always make sense to pay for interpreter startup overhead on every request (EDIT: this is not the case according to the Fission developers), especially not now that compiled code is so easy.

timdorr 10y ago
They're talking about cold start, not long-term performance. After the initial warm up period, performance should be as fast as Node/Python can execute it. It's the same experience for Lambda and Cloud Functions in my experience.
EwanToo 10y ago
The cold start impact is quite high, but that's only for APIs which have been idle or are scaling up rapidly. If you were getting even 1 request per second, the container last used would remain live, so the response time would then be in line with normal responses from a typical NodeJS server running the same code. I agree though, supplying binaries in something like Go or Rust will be a very nice addition.
soamv 10y ago
[fission developer here] About that 100msec overhead -- there's a bunch of optimizations yet to be done, and also we think we can reduce the median at the expense of tail latency; but for now 100msec is good enough performance for a good enough set of use cases (but of course we'd love to hear from people for whom it really doesn't work at all). Also, we could add knobs that let you tune how long a function stays "warm", so you can tune the latency overhead vs. resource consumption trade-off. > But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. We're very interested in this use case. We're redesigning the function environment stuff right now to better support compiled languages, and allowing users to bypass the source-code level stuff is quite possible. BTW, to be clear, for a compiled language Fission wouldn't compile source code at runtime; there would be separate build stage that would run when you add the function to fission. I'm also not sure why you say we pay interpreter startup overhead on every request (we don't).
runeks 10y ago
> But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. Isn't that just what "regular" Kubernetes is, except that you wrap a Linux image/container around your static ELF binary? The Stack build tool for Haskell has an option for building as a Docker image, wrapped around the static Linux ELF binary that it would otherwise produce. I assume something similar exists for Go and Rust.
- TheIronYuppie 10y ago
  Yep, this is exactly right. The tradeoff here (and it's a good one) is you don't require any binaries at all. Disclosure: I work at Google on Kubernetes.
Mizza 10y ago
Warm Lambdas (using Zappa's auto keep-warm) are actually 20-40ms, which makes them suitable for web servers.
- rads 10y ago
  In case anyone else was wondering how the auto keep-warm feature is implemented: it's a CloudWatch Event that runs the Lambda every four minutes by default. While this probably keeps the majority of the requests warm, I would expect that there would still be some cold ones every now and then. Amazon doesn't provide a guarantee in how the containers get recycled. This is just me speculating, though.
  - adjohn 10y ago
    We did some analysis[1] on cold starts in AWS Lambda, and found that there is a cold start if there's no activity for 5 minutes, or regardless of activity, there will be a cold start every 4 hours from creation of the underlying VM. [1] - https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/ https://www.iopipe.com/2016/09/understanding-aws-lambda-cold...
    - rads 10y ago
      Thanks, I was hoping someone would come along with some real data!
shanemhansen 10y ago
Everything old is new again. So in the old days there was a way to supply a binary rather than source code. That binary could be fast starting. There was even an open standard for connecting those binaries to different web servers. The standard used unix patterns such as reading stdin and writing stdout. Sometimes it seems like people have created an insanely high learning curve to run their CGI programs. Sometimes it seems like most of the magic and value add of some of these platforms is that they reduce some of the overhead they added.
- coredog64 10y ago
  Someone needs to start a "container global initiative", and then we can keep all of our binary containers in "cgi-bin".
extrapickles 10y ago
I always thought that things like Fission are best for things that will take long enough that 200ms doesn't make much of a difference such as video encoding, updating recommendations, etc. What reducing the overhead does is make it useful for more things like machine vision for images (eg: is this user uploaded image safe for work) but its still nowhere near good enough for doing a DB query that takes 1ms or serving static content.

by ekidd 10y ago

timdorr 10y ago

They're talking about cold start, not long-term performance. After the initial warm up period, performance should be as fast as Node/Python can execute it. It's the same experience for Lambda and Cloud Functions in my experience.

EwanToo 10y ago

The cold start impact is quite high, but that's only for APIs which have been idle or are scaling up rapidly. If you were getting even 1 request per second, the container last used would remain live, so the response time would then be in line with normal responses from a typical NodeJS server running the same code. I agree though, supplying binaries in something like Go or Rust will be a very nice addition.

soamv 10y ago

[fission developer here] About that 100msec overhead -- there's a bunch of optimizations yet to be done, and also we think we can reduce the median at the expense of tail latency; but for now 100msec is good enough performance for a good enough set of use cases (but of course we'd love to hear from people for whom it really doesn't work at all). Also, we could add knobs that let you tune how long a function stays "warm", so you can tune the latency overhead vs. resource consumption trade-off. > But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. We're very interested in this use case. We're redesigning the function environment stuff right now to better support compiled languages, and allowing users to bypass the source-code level stuff is quite possible. BTW, to be clear, for a compiled language Fission wouldn't compile source code at runtime; there would be separate build stage that would run when you add the function to fission. I'm also not sure why you say we pay interpreter startup overhead on every request (we don't).

runeks 10y ago

> But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. Isn't that just what "regular" Kubernetes is, except that you wrap a Linux image/container around your static ELF binary? The Stack build tool for Haskell has an option for building as a Docker image, wrapped around the static Linux ELF binary that it would otherwise produce. I assume something similar exists for Go and Rust.

TheIronYuppie 10y ago
Yep, this is exactly right. The tradeoff here (and it's a good one) is you don't require any binaries at all. Disclosure: I work at Google on Kubernetes.

Mizza 10y ago

Warm Lambdas (using Zappa's auto keep-warm) are actually 20-40ms, which makes them suitable for web servers.

rads 10y ago
In case anyone else was wondering how the auto keep-warm feature is implemented: it's a CloudWatch Event that runs the Lambda every four minutes by default. While this probably keeps the majority of the requests warm, I would expect that there would still be some cold ones every now and then. Amazon doesn't provide a guarantee in how the containers get recycled. This is just me speculating, though.
- adjohn 10y ago
  We did some analysis[1] on cold starts in AWS Lambda, and found that there is a cold start if there's no activity for 5 minutes, or regardless of activity, there will be a cold start every 4 hours from creation of the underlying VM. [1] - https://www.iopipe.com/2016/09/understanding-aws-lambda-coldstarts/ https://www.iopipe.com/2016/09/understanding-aws-lambda-cold...
  - rads 10y ago
    Thanks, I was hoping someone would come along with some real data!

shanemhansen 10y ago

Everything old is new again. So in the old days there was a way to supply a binary rather than source code. That binary could be fast starting. There was even an open standard for connecting those binaries to different web servers. The standard used unix patterns such as reading stdin and writing stdout. Sometimes it seems like people have created an insanely high learning curve to run their CGI programs. Sometimes it seems like most of the magic and value add of some of these platforms is that they reduce some of the overhead they added.

coredog64 10y ago
Someone needs to start a "container global initiative", and then we can keep all of our binary containers in "cgi-bin".

extrapickles 10y ago

I always thought that things like Fission are best for things that will take long enough that 200ms doesn't make much of a difference such as video encoding, updating recommendations, etc. What reducing the overhead does is make it useful for more things like machine vision for images (eg: is this user uploaded image safe for work) but its still nowhere near good enough for doing a DB query that takes 1ms or serving static content.