7 ms·

A 1 KB Docker Container

216 points by nathan-osman 9y ago

maruhan2 9y ago
Im not familiar with dockers so bear with me, but why is another container needed for the reverse proxy?
- nathan-osman 9y ago
  The reverse proxy is configured to route requests to any container that is currently running. For example, if I have a Jenkins container, as long as it is running, the reverse proxy will send requests to it. In this particular case, I need to proxy a remote host. In order to do that, I have a container running with the appropriate labels, the proxy sees the container running, and it proxies requests to the remote host.
  - davman 9y ago
    I think I'm being particularly dense here, but do you mean the proxy is proxying to your tiny 1kb container? If so, what happens to the traffic? Or because the tiny container is running the proxy proxies somewhere else? i.e you're using the presence of a container with a label as configuration for the proxy?
    - nathan-osman 9y ago
      I know, it's a bit difficult to explain and a tad confusing. No, the proxy is not sending any traffic to the container. Rather, the container, by simple virtue of being in the running state, tells the proxy to send requests to the domain specified by the label on the container.
      - deleted 9y ago
        [deleted]
      - djhworld 9y ago
        What happens if the container isn't running?
        nathan-osman 9y ago
        Then the proxy will not route requests to the host specified by the container's label.
        radarsat1 9y ago
        You're using the running state flag of a container as an IPC mechanism?
        nathan-osman 9y ago
        Essentially, yes. And the labels.
avian 9y ago
Another example of a really small container that doesn't do much, but is made without using assembly directly is the Docker "hello-world". It's built from C without linking in libc: https://github.com/docker-library/hello-world/blob/master/hello.c https://github.com/docker-library/hello-world/blob/master/he... I always thought this was a bit misleading. A "hello world" container is 1 kB, but the bare minimum container that does something useful in practice is rarely less than 100 MB in size.
- nathan-osman 9y ago
  Interesting. I must confess, I didn't realize there was a way to make syscalls directly from C without resorting to inline assembly.
- raesene2 9y ago
  If you base off alpine, you can get useful containers quite a lot smaller than 100MB. One example i use is an agent i deploy to kubernetes clusters to do some security scanning. The scripts are ruby and the image clocks in at 9MB compressed https://hub.docker.com/r/raesene/kaa-agent/tags/ https://hub.docker.com/r/raesene/kaa-agent/tags/
  - nathan-osman 9y ago
    Yes, I use Alpine for a lot of my other containers. I love the simplicity of the package manager as well.
    - theptip 9y ago
      Alpine's package manager has the great property that you don't need to update the index in order to fetch a package IIRC; the whole `apt-get update && apt-get install && <cleanup apt-cache>` dance is quite tedious in debian-based Docker containers.
      - colemickens 9y ago
        No, you still need to, but there's a compact syntax for it that will update and discard the index in a single 'add' command. It's unavoidable - somewhere some querying is happening in order map the package name/ver to a download link.
ttwwmm 9y ago
If you want to reduce the size more, try emitting a single layer as a tarball which you can pipe to docker load. That will reduce your layer count to one, plus you can omit a bunch of the metadata that Docker includes when you build from a Dockerfile (it doesn't seem to care at runtime). https://github.com/moby/moby/blob/master/image/spec/v1.md https://github.com/moby/moby/blob/master/image/spec/v1.md
- raesene6 9y ago
  Indeed. Another useful technique, which has been recently introduced, is Multi-Stage Builds https://docs.docker.com/engine/userguide/eng-image/multistage-build/ https://docs.docker.com/engine/userguide/eng-image/multistag... . This lets you avoid putting tools needed for compilation into the final image.
brobinson 9y ago
I have one that is 344 bytes: https://github.com/dseevr/cpu_consumer https://github.com/dseevr/cpu_consumer It just runs a tight loop that consumes an entire core.
- chanux 9y ago
  This is cool! thanks.
ffk 9y ago
The smallest useful container I know of is 129B. It was created to test how many containers docker can spin up while reducing the overhead of what was in the container itself. tianon/sleeping-beauty latest 2e8193709fa7 6 months ago 129B https://github.com/tianon/dockerfiles/tree/master/sleeping-beauty https://github.com/tianon/dockerfiles/tree/master/sleeping-b...
- logicallee 9y ago
  Interesting. Incidentally, > It was created to test how many containers docker can spin up What was the answer? I'd think on the order of 200-300 on a server with 64 GB of RAM. (Pure guess!)
  - oso2k 9y ago
    That is very low. The current official limitation in OpenShift (a Kubernetes distro) is 250 but there have been lab clusters which have gone higher.
    - plag 9y ago
      There are articles about running 2500 web server containers on a raspberry pi :)
  - TheDong 9y ago
    Depends on several other variables. If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured. If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container. "containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most. The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k. Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total. Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.
primeblue 9y ago
Great stuff
deleted 9y ago
[deleted]
throwme_1980 9y ago
What problem are we trying to solve here ? And I don't mean the work around for whatever the other hacky application's flaw. What value does this give me?
pbiggar 9y ago
I did a similar thing, using a tiny executable that someone else had made. However, even though the container was around 100 bytes, I couldn't make a container that was smaller than some much larger number, maybe 512kb? This was in 2015, so maybe that limit has changed - any docker folks know anything about this?
nemasu 9y ago
Reminded me of people making docker containers out of this: https://github.com/nemasu/asmttpd https://github.com/nemasu/asmttpd eg. https://hub.docker.com/r/0xff/asmttpd/ https://hub.docker.com/r/0xff/asmttpd/ 7KB web server container.
- kozak 9y ago
  Are there any non-academic reasons to do use such servers?
  - pletnes 9y ago
    Embedded stuff like routers? Serving a user manual for some network connected thing with a microcontroller?
thebsdbox 9y ago
As part of a competition before the last DockerCon I managed to get a container down to 69B Details: http://thebsdbox.co.uk/in-pursuit-of-a-tinier-binary-er/ http://thebsdbox.co.uk/in-pursuit-of-a-tinier-binary-er/ Code: https://gist.github.com/thebsdbox/29e395299f89b52214b66269f5b33f7d https://gist.github.com/thebsdbox/29e395299f89b52214b66269f5...
- majewsky 9y ago
  Is "69 bytes" for the binary, or for the whole container image? I would expect at least some metadata.
- rhizome 9y ago
  Nice.
- frameloss 9y ago
  Nice! I did something similar a few years back as a bit of a joke at work, and best I could come up with was 345 bytes ... mov rax,34 ; pause() syscall
userbinator 9y ago
Notice that the executable itself contains less than 100 bytes of instructions, but the file is still 736 bytes. Even without optimising the Asm itself (I can see at least 2-3 bytes improvement at a glance), that could probably be reduced even further: http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm... 736B may seem tiny to most people, but if you're working in Asm that's a lot --- there's plenty of interesting things (beyond "call the OS a few times") the demoscene has done with smaller binaries; here's an assortment of 512B ones: http://www.pouet.net/prodlist.php?type[]=512b http://www.pouet.net/prodlist.php?type[]=512b
adlpz 9y ago
This is of course pretty pointless, I mean it's neat but all these tiny containers don't really do anything so it isn't much more than a cool trick. However there is a great lesson to take from this: you can create single-binary but really useful containers for just a few MBs, which is nothing in practice for a usual sized server, an a lot more lightweight than usual containers based off Alpine or Ubuntu. In fact I run my static websites using that: a small 8-ish MB statically compiled web server "written" (it's really just library glue) in Go.
- j_s 9y ago
  Do you put the static content into the container?
  - nathan-osman 9y ago
    I've done this with a couple of applications, using something like fileb0x in the build process to convert the files into Go source which can then be compiled into the final executable.
siscia 9y ago
I developed a little side project that aimed to be a serveless open source alternative, it is called effe and uses go as main language. The whole idea is to compile a go program and put it into a docker container and have it listening to the network for a single HTTP endpoint. It is interesting because an useful images come to be less than 6MB :) Link to the project: https://github.com/siscia/effe-tool https://github.com/siscia/effe-tool
Varcht 9y ago
OT, what's the smallest windows/.net containers out there? I have some legacy stuff I'd like to dockerize but they end up so big and slow to deploy. I've just been rewriting them.
darren0 9y ago
Container in a (240 character) tweet https://twitter.com/thaJeztah/status/913378165124423680 https://twitter.com/thaJeztah/status/913378165124423680 Replace -d with -D for macOS
jzelinskie 9y ago
For reference, Kubernetes uses a `pause`[0] for a similar reason. [0]: https://github.com/kubernetes/kubernetes/tree/master/build/pause https://github.com/kubernetes/kubernetes/tree/master/build/p...
arianvanp 9y ago
We already have a portable container format for executables with no dependencies... It's called an ELF binary... Why would you even put a static binary in a container in the first place? I don't get it.
- mixmastamyk 9y ago
  You still need to administer the box it runs on. Offloading that to others is often useful.
  - arianvanp 9y ago
    I guess you mean something like Compose or Kubernetes. I think it's weird these things force you to use the docker runtime, personally. I would really love a more flexible definition in kubernetes of what a "resource to be ran" means. I know they support multiple container runtimes now, but what if I dont want a container runtime at all? Nomad supports raw executables to be downloaded and scheduled, which is nice(https://nomadproject.io https://nomadproject.io) but then again, kubernetes seems miles ahead in what it supports (autoscaling, volume claims, RBAC etc) Otherwise, more traditional means of managing your services can be employed. I've got a lot of leverage out of systemd myself. Which by the way, supports all the features of a proper container runtime. You can namespace your executable, Chroot it, limit what devices it can access, etc, which is kinda awesome. Check out `man systemd.exec` and `man systemd.resource-control`
    - rckrd 9y ago
      Kubernetes absolutely does not force you to use the docker runtime. In fact, there has been a lot of work to avoid this by creating the CRI[0]. Kubernetes also supports extensions like the Third Party Resource or their successor, Custom Resource Definitions. KubeVirt[1] is an example of extending resources to include VMs [0] http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html http://blog.kubernetes.io/2016/12/container-runtime-interfac... [1] https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt
daviesgeek 9y ago
This was a really fascinating read. Enjoyed it thoroughly

216 points by nathan-osman 9y ago

maruhan2 9y ago

Im not familiar with dockers so bear with me, but why is another container needed for the reverse proxy?

nathan-osman 9y ago
The reverse proxy is configured to route requests to any container that is currently running. For example, if I have a Jenkins container, as long as it is running, the reverse proxy will send requests to it. In this particular case, I need to proxy a remote host. In order to do that, I have a container running with the appropriate labels, the proxy sees the container running, and it proxies requests to the remote host.
- davman 9y ago
  I think I'm being particularly dense here, but do you mean the proxy is proxying to your tiny 1kb container? If so, what happens to the traffic? Or because the tiny container is running the proxy proxies somewhere else? i.e you're using the presence of a container with a label as configuration for the proxy?
  - nathan-osman 9y ago
    I know, it's a bit difficult to explain and a tad confusing. No, the proxy is not sending any traffic to the container. Rather, the container, by simple virtue of being in the running state, tells the proxy to send requests to the domain specified by the label on the container.
    - deleted 9y ago
      [deleted]
    - djhworld 9y ago
      What happens if the container isn't running?
      - nathan-osman 9y ago
        Then the proxy will not route requests to the host specified by the container's label.
        radarsat1 9y ago
        You're using the running state flag of a container as an IPC mechanism?
        nathan-osman 9y ago
        Essentially, yes. And the labels.

avian 9y ago

Another example of a really small container that doesn't do much, but is made without using assembly directly is the Docker "hello-world". It's built from C without linking in libc: https://github.com/docker-library/hello-world/blob/master/hello.c https://github.com/docker-library/hello-world/blob/master/he... I always thought this was a bit misleading. A "hello world" container is 1 kB, but the bare minimum container that does something useful in practice is rarely less than 100 MB in size.

nathan-osman 9y ago
Interesting. I must confess, I didn't realize there was a way to make syscalls directly from C without resorting to inline assembly.
raesene2 9y ago
If you base off alpine, you can get useful containers quite a lot smaller than 100MB. One example i use is an agent i deploy to kubernetes clusters to do some security scanning. The scripts are ruby and the image clocks in at 9MB compressed https://hub.docker.com/r/raesene/kaa-agent/tags/ https://hub.docker.com/r/raesene/kaa-agent/tags/
- nathan-osman 9y ago
  Yes, I use Alpine for a lot of my other containers. I love the simplicity of the package manager as well.
  - theptip 9y ago
    Alpine's package manager has the great property that you don't need to update the index in order to fetch a package IIRC; the whole `apt-get update && apt-get install && <cleanup apt-cache>` dance is quite tedious in debian-based Docker containers.
    - colemickens 9y ago
      No, you still need to, but there's a compact syntax for it that will update and discard the index in a single 'add' command. It's unavoidable - somewhere some querying is happening in order map the package name/ver to a download link.

ttwwmm 9y ago

If you want to reduce the size more, try emitting a single layer as a tarball which you can pipe to docker load. That will reduce your layer count to one, plus you can omit a bunch of the metadata that Docker includes when you build from a Dockerfile (it doesn't seem to care at runtime). https://github.com/moby/moby/blob/master/image/spec/v1.md https://github.com/moby/moby/blob/master/image/spec/v1.md

raesene6 9y ago
Indeed. Another useful technique, which has been recently introduced, is Multi-Stage Builds https://docs.docker.com/engine/userguide/eng-image/multistage-build/ https://docs.docker.com/engine/userguide/eng-image/multistag... . This lets you avoid putting tools needed for compilation into the final image.

brobinson 9y ago

I have one that is 344 bytes: https://github.com/dseevr/cpu_consumer https://github.com/dseevr/cpu_consumer It just runs a tight loop that consumes an entire core.

chanux 9y ago
This is cool! thanks.

ffk 9y ago

The smallest useful container I know of is 129B. It was created to test how many containers docker can spin up while reducing the overhead of what was in the container itself. tianon/sleeping-beauty latest 2e8193709fa7 6 months ago 129B https://github.com/tianon/dockerfiles/tree/master/sleeping-beauty https://github.com/tianon/dockerfiles/tree/master/sleeping-b...

logicallee 9y ago
Interesting. Incidentally, > It was created to test how many containers docker can spin up What was the answer? I'd think on the order of 200-300 on a server with 64 GB of RAM. (Pure guess!)
- oso2k 9y ago
  That is very low. The current official limitation in OpenShift (a Kubernetes distro) is 250 but there have been lab clusters which have gone higher.
  - plag 9y ago
    There are articles about running 2500 web server containers on a raspberry pi :)
- TheDong 9y ago
  Depends on several other variables. If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured. If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container. "containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most. The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k. Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total. Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.

primeblue 9y ago

Great stuff

deleted 9y ago

[deleted]

throwme_1980 9y ago

What problem are we trying to solve here ? And I don't mean the work around for whatever the other hacky application's flaw. What value does this give me?

pbiggar 9y ago

I did a similar thing, using a tiny executable that someone else had made. However, even though the container was around 100 bytes, I couldn't make a container that was smaller than some much larger number, maybe 512kb? This was in 2015, so maybe that limit has changed - any docker folks know anything about this?

nemasu 9y ago

Reminded me of people making docker containers out of this: https://github.com/nemasu/asmttpd https://github.com/nemasu/asmttpd eg. https://hub.docker.com/r/0xff/asmttpd/ https://hub.docker.com/r/0xff/asmttpd/ 7KB web server container.

kozak 9y ago
Are there any non-academic reasons to do use such servers?
- pletnes 9y ago
  Embedded stuff like routers? Serving a user manual for some network connected thing with a microcontroller?

thebsdbox 9y ago

As part of a competition before the last DockerCon I managed to get a container down to 69B Details: http://thebsdbox.co.uk/in-pursuit-of-a-tinier-binary-er/ http://thebsdbox.co.uk/in-pursuit-of-a-tinier-binary-er/ Code: https://gist.github.com/thebsdbox/29e395299f89b52214b66269f5b33f7d https://gist.github.com/thebsdbox/29e395299f89b52214b66269f5...

majewsky 9y ago
Is "69 bytes" for the binary, or for the whole container image? I would expect at least some metadata.
rhizome 9y ago
Nice.
frameloss 9y ago
Nice! I did something similar a few years back as a bit of a joke at work, and best I could come up with was 345 bytes ... mov rax,34 ; pause() syscall

userbinator 9y ago

Notice that the executable itself contains less than 100 bytes of instructions, but the file is still 736 bytes. Even without optimising the Asm itself (I can see at least 2-3 bytes improvement at a glance), that could probably be reduced even further: http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html http://www.muppetlabs.com/~breadbox/software/tiny/teensy.htm... 736B may seem tiny to most people, but if you're working in Asm that's a lot --- there's plenty of interesting things (beyond "call the OS a few times") the demoscene has done with smaller binaries; here's an assortment of 512B ones: http://www.pouet.net/prodlist.php?type[]=512b http://www.pouet.net/prodlist.php?type[]=512b

adlpz 9y ago

This is of course pretty pointless, I mean it's neat but all these tiny containers don't really do anything so it isn't much more than a cool trick. However there is a great lesson to take from this: you can create single-binary but really useful containers for just a few MBs, which is nothing in practice for a usual sized server, an a lot more lightweight than usual containers based off Alpine or Ubuntu. In fact I run my static websites using that: a small 8-ish MB statically compiled web server "written" (it's really just library glue) in Go.

j_s 9y ago
Do you put the static content into the container?
- nathan-osman 9y ago
  I've done this with a couple of applications, using something like fileb0x in the build process to convert the files into Go source which can then be compiled into the final executable.

siscia 9y ago

I developed a little side project that aimed to be a serveless open source alternative, it is called effe and uses go as main language. The whole idea is to compile a go program and put it into a docker container and have it listening to the network for a single HTTP endpoint. It is interesting because an useful images come to be less than 6MB :) Link to the project: https://github.com/siscia/effe-tool https://github.com/siscia/effe-tool

Varcht 9y ago

OT, what's the smallest windows/.net containers out there? I have some legacy stuff I'd like to dockerize but they end up so big and slow to deploy. I've just been rewriting them.

darren0 9y ago

Container in a (240 character) tweet https://twitter.com/thaJeztah/status/913378165124423680 https://twitter.com/thaJeztah/status/913378165124423680 Replace -d with -D for macOS

jzelinskie 9y ago

For reference, Kubernetes uses a `pause`[0] for a similar reason. [0]: https://github.com/kubernetes/kubernetes/tree/master/build/pause https://github.com/kubernetes/kubernetes/tree/master/build/p...

arianvanp 9y ago

We already have a portable container format for executables with no dependencies... It's called an ELF binary... Why would you even put a static binary in a container in the first place? I don't get it.

mixmastamyk 9y ago
You still need to administer the box it runs on. Offloading that to others is often useful.
- arianvanp 9y ago
  I guess you mean something like Compose or Kubernetes. I think it's weird these things force you to use the docker runtime, personally. I would really love a more flexible definition in kubernetes of what a "resource to be ran" means. I know they support multiple container runtimes now, but what if I dont want a container runtime at all? Nomad supports raw executables to be downloaded and scheduled, which is nice(https://nomadproject.io https://nomadproject.io) but then again, kubernetes seems miles ahead in what it supports (autoscaling, volume claims, RBAC etc) Otherwise, more traditional means of managing your services can be employed. I've got a lot of leverage out of systemd myself. Which by the way, supports all the features of a proper container runtime. You can namespace your executable, Chroot it, limit what devices it can access, etc, which is kinda awesome. Check out `man systemd.exec` and `man systemd.resource-control`
  - rckrd 9y ago
    Kubernetes absolutely does not force you to use the docker runtime. In fact, there has been a lot of work to avoid this by creating the CRI[0]. Kubernetes also supports extensions like the Third Party Resource or their successor, Custom Resource Definitions. KubeVirt[1] is an example of extending resources to include VMs [0] http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html http://blog.kubernetes.io/2016/12/container-runtime-interfac... [1] https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt

daviesgeek 9y ago

This was a really fascinating read. Enjoyed it thoroughly