4 ms·

Why would a container be slow to create? It's effectively a fork with namespacing options. The overlay file system is just a product of one or more layers that

Why would a container be slow to create? It's effectively a fork with namespacing options. The overlay file system is just a product of one or more layers that are already computed; there's not much complexity to its mounting compared to something like ext4 or NFS. I haven't timed it, but containers seem damn fast to start with Docker and Kubernetes.

TheDong 9y ago
> I haven't timed it, but containers seem damn fast to start with Docker and Kubernetes. You should time it then, it's incredibly slow these days $ time docker run busybox /bin/true .388s total $ time docker run --net=none busybox /bin/true .310s total The above times, of over 300ms, are with docker 17.06 and the overlay2 driver on a powerful computer with an ssd. It's horrifically slow because the entire docker codebase is disgustingly bloated. It has to fork something like 10 processes and synchronize between them (for everything from network initialization to runc hooks). It has to make ipc calls from docker-cli to dockerd to containerd and back again, allocating and freeing large chunks of memory for grpc along the way. It's incredibly slow. On the other hand, just forking and running something with namespacing options is effectively instant: $ dir=$(mktemp -d) $ docker run --name bbexport busybox /bin/true $ cd $dir $ docker export bbexport | tar x $ time nsenter --root=$dir /bin/true 0.000s total This is the reason they don't let you use arbitrary docker containers; docker containers are a ton more than a fork and some options. They're a bloated moving target. What lambda does allows them to optmize much more. In fact, lambda doesn't even have to fork a process sometimes because they can keep the e.g. javascript runtime running and then just add a new request into the existing process directly. They can freeze and thaw environments from known good states for request processing. But most importantly, by not being compatible in any way with docker, they don't have to deal with the bloated mess that is docker and the ecosystem around it. They don't have to load 3 daemons measuring 50MB into memory to run 100kb of javascript (which they can currently run more quickly than docker can even start the simpleist container)
jitl 9y ago
Even though a container starts "damn fast" in Docker, you're still adding significant latency to each call of your function if it requires a new container. $ time docker run alpine:3.6 sh -c exit docker run alpine:3.6 sh -c exit 0.02s user 0.02s system 3% cpu 1.191 total $ time sh -c exit sh -c exit 0.00s user 0.01s system 94% cpu 0.007 total Then add to that the cost of loading your function's code into memory. For an interpreted language, that is probably going to take a while as the runtime does a bunch of system calls and parses text files. Even for a compiled, statically-linked language like Go, the executable (a random binary I picked was 50MB) needs to be copied into RAM. For this reason, Lambda tries to keep your containers/function-wrapper-applications booted up in RAM: https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/ https://aws.amazon.com/blogs/compute/container-reuse-in-lamb...
- lobster_johnson 9y ago
  True, but that's Docker. There's nothing, that I know, that would make Linux containers inherently much slower than a fork. Amazon could conceivably also write their own container runtime that kept the container ready to run without actually starting the app (though I'm sure there's a lot more to it -- I don't know how Lambda is implemented).

TheDong 9y ago

> I haven't timed it, but containers seem damn fast to start with Docker and Kubernetes. You should time it then, it's incredibly slow these days $ time docker run busybox /bin/true .388s total $ time docker run --net=none busybox /bin/true .310s total The above times, of over 300ms, are with docker 17.06 and the overlay2 driver on a powerful computer with an ssd. It's horrifically slow because the entire docker codebase is disgustingly bloated. It has to fork something like 10 processes and synchronize between them (for everything from network initialization to runc hooks). It has to make ipc calls from docker-cli to dockerd to containerd and back again, allocating and freeing large chunks of memory for grpc along the way. It's incredibly slow. On the other hand, just forking and running something with namespacing options is effectively instant: $ dir=$(mktemp -d) $ docker run --name bbexport busybox /bin/true $ cd $dir $ docker export bbexport | tar x $ time nsenter --root=$dir /bin/true 0.000s total This is the reason they don't let you use arbitrary docker containers; docker containers are a ton more than a fork and some options. They're a bloated moving target. What lambda does allows them to optmize much more. In fact, lambda doesn't even have to fork a process sometimes because they can keep the e.g. javascript runtime running and then just add a new request into the existing process directly. They can freeze and thaw environments from known good states for request processing. But most importantly, by not being compatible in any way with docker, they don't have to deal with the bloated mess that is docker and the ecosystem around it. They don't have to load 3 daemons measuring 50MB into memory to run 100kb of javascript (which they can currently run more quickly than docker can even start the simpleist container)

jitl 9y ago

Even though a container starts "damn fast" in Docker, you're still adding significant latency to each call of your function if it requires a new container. $ time docker run alpine:3.6 sh -c exit docker run alpine:3.6 sh -c exit 0.02s user 0.02s system 3% cpu 1.191 total $ time sh -c exit sh -c exit 0.00s user 0.01s system 94% cpu 0.007 total Then add to that the cost of loading your function's code into memory. For an interpreted language, that is probably going to take a while as the runtime does a bunch of system calls and parses text files. Even for a compiled, statically-linked language like Go, the executable (a random binary I picked was 50MB) needs to be copied into RAM. For this reason, Lambda tries to keep your containers/function-wrapper-applications booted up in RAM: https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/ https://aws.amazon.com/blogs/compute/container-reuse-in-lamb...

lobster_johnson 9y ago
True, but that's Docker. There's nothing, that I know, that would make Linux containers inherently much slower than a fork. Amazon could conceivably also write their own container runtime that kept the container ready to run without actually starting the app (though I'm sure there's a lot more to it -- I don't know how Lambda is implemented).