5 ms·

The smallest useful container I know of is 129B. It was created to test how many containers docker can spin up while reducing the overhead of what was in the co

• logicallee 9y ago
  Interesting. Incidentally, > It was created to test how many containers docker can spin up What was the answer? I'd think on the order of 200-300 on a server with 64 GB of RAM. (Pure guess!)
  • oso2k 9y ago
    That is very low. The current official limitation in OpenShift (a Kubernetes distro) is 250 but there have been lab clusters which have gone higher.
    • plag 9y ago
      There are articles about running 2500 web server containers on a raspberry pi :)
  • TheDong 9y ago
    Depends on several other variables. If you use default docker options, you'll be creating a veth pair per container. You might run into a limit there at around 1024 containers. You also might hit ulimit if your system isn't well configured. If you use --net=none, you won't hit that issue, and you'll probably be able to manage quite a few The resource usage ends up being roughly 4 bytes rss for the executable in the container and around 3.5MB for the "containerd-shim" go binary that parents the container. "containerd" and "dockerd" both probably have a little extra resource usage per container they're managing, but I'd guess that's on the order of about 200KB per at most. The next big limit you'll hit is the process/pid limit (/proc/sys/kernel/pid_max) which defaults to 32k. Fortunately, due to the memory overhead of a bit under 4MB, you probably won't get there on your 64GB of ram server and might cap out at around 15k containers total. Experimentally, my linux laptop (running docker 17.06) is able to run 1100 copies of that sleeping-beuaty container using almost exactly 2GB RSS additional memory and no noticeable additional cpu This is even better than I calculated above, possibly due to shared memory for containerd-shim. I'm not investigating further.
    • e12e 9y ago
      What happens when you hit the limit? Thrashing swap or oom kills? (And what kind of overcommit policy did you use?)
      • TheDong 9y ago
        I didn't hit any limit, but since I don't have swap it would likely be oom kills. The first part of my post is speculating, the second part I ran an arbitrary number and observed resource usage to allow extrapolation, but didn't hit a limit.
        • ffk 9y ago
          Thanks for reproducing the experiment. :) I sent a message to tianon asking him if he remembers what the original numbers were. He told me long ago, I think it was around 1000. This was well before the go shim or dockerd existed.
    • logicallee 9y ago
      thanks, this was interesting! That's better than I expected.
  • zokier 9y ago
    I would expect a 64 gig machine to be able to host 200-300 VMs, and an order or two magnitudes more containers
    • dijit 9y ago
      I guess you don't have anything CPU intensive running. General good practice for sysadmins is to affine CPU cores/threads so that you don't end up flushing the CPU cache too aggressively and you don't split your VM's over NUMA zones because memory locality is important.
    • logicallee 9y ago
      Well, I was asking what the actual result was. GP (who didn't respond to me) mentioned it was an experiment. TheDong gave experimental results in a reply though.
  • nathan-osman 9y ago
    The server hosting the article above is also running 38 other containers - databases, WordPress blogs, Jenkins (CI), Gogs (Git), etc. And it only has 4 GB of RAM.
    • korzun 9y ago
      Doesn't mean it's a good idea; Jenkins recommends 2GB. That's 50% right there.
  • ffk 9y ago
    I'll have to ask tianon to see what he says. I don't remember the number.
    • logicallee 9y ago
      this thread is kind of stale - but if you'd do so and reply (you can also email me at the email in my profile) I'll appreciate it.