17 ms·

Docker and the PID 1 zombie reaping problem

119 points by ohcoder 12y ago

justincormack 12y ago
Surely writing a minimal init in C (or Go if you like) is better than using Python... it is not a lot of code, and frankly I would trust it not to die and kill your container more. Plus Python has a chunk of dependencies you don't need in a minimal container.
- FooBarWidget 12y ago
  Lots of system tools are written in Python nowadays. If you use a Debian, Ubuntu or CentOS base image then there's a good chance that Python is preinstalled because some system tool depends on it. It was also much easier to write the tool in Python than in C. Writing it in C would have taken much longer, and at the end of the day we wanted to get work done instead of worrying about a few MBs, which only take 1 second to download and which cost $0.001 of disk space.
  - mborch 12y ago
    It's trivial to write this program in C and it is a waste of memory to load up 50 MB of Python just to accomplish this simple task.
    - FooBarWidget 12y ago
      Actually, the Python process eats less than 5 MB of RAM. Most other stuff on the system eats more RAM. Writing my_init took much more time than you think, and I do not regret choosing Python. My_init not only handles process reaping, but also environment variable handling and bunch of other stuff. Writing text parsers in C is a huge pain (libc has no regexps that are as easy to use as Python). I do not oppose rewriting it in C, but given limited manpower and the tiny benefits it's not high on my todo list either. If you are worried about resource usage, then I would very much welcome contributions for rewriting it in C.
      - mborch 12y ago
        Fair enough, but for the record it's at least 10 MB (try starting up Python and import your list of modules). The environment variable handling and other stuff could perhaps be handled in a separate bash-script (or Python), such that the scope of the actual reaper is sufficiently small (to use a lower-level language).
        kstenerud 12y ago
        5-10MB is a big nothing, unless you're talking about running hundreds of images that consume 20-30mb each. in which case you would have some custom image structure to minimize everything anyway. But then you'd want to consider the infrastructure development cost vs just buying more iron. There comes a point where you have to take a look at the diminishing returns and say "That's good enough."
      - pdw 12y ago
        libc does have regexps. http://pubs.opengroup.org/onlinepubs/009695399/functions/regcomp.html http://pubs.opengroup.org/onlinepubs/009695399/functions/reg...
      - edwintorok 12y ago
        Why not use already existing init? runit seems small enough: http://smarden.org/runit/ http://smarden.org/runit/ Or if you want small have a look here for minimal PID 1 "So how should init be done right?" http://ewontfix.com/14/ http://ewontfix.com/14/
        FooBarWidget 12y ago
        Baseimage-docker does use Runit. my_init calls Runit.
        edwintorok 12y ago
        Why not use http://smarden.org/runit/runit-init.8.html http://smarden.org/runit/runit-init.8.html and run my_init from runit?
  - falcolas 12y ago
    Writing it in Python also adds a lot of dependencies and bloat to the docker image which don't really need to be there. Particularly when there are simple init scripts like [1] which could be easily adapted to fill both the reaping and restart problem. [1] http://git.suckless.org/sinit/tree/sinit.c http://git.suckless.org/sinit/tree/sinit.c
  - justincormack 12y ago
    It really does not need much code in C, here is a simple init https://github.com/arachsys/init/blob/master/init.c https://github.com/arachsys/init/blob/master/init.c for example. In particular you want to avoid memory allocation so it cannot fail even in low memory situations, which you cannot guarantee with Python. That means you might die reaping children with low memory which would kill your container.
- dnr 12y ago
  Indeed: I solved this minor issue with a 23 line C program that compiles to a 8766 byte binary. It's really not a big deal. (Mine doesn't pass on signals, but that's not important for us.)
tjholowaychuk 12y ago
Isn't this effectively a non-issue? It's literally just a tiny entry in a table that is held no? Or does it just cause Docker to behave oddly?
- FooBarWidget 12y ago
  There are two issues. First is Murphey's law. 1 zombie process is probably harmless, but if the amount grows then you'll fill up kernel limits and the container cannot create further processes. Second, zombie processes that don't go away can also interfere with software that check for the existence of processes. For example, [the Phusion Passenger application server](https://www.phusionpassenger.com/ https://www.phusionpassenger.com/) manages processes. It restarts processes when they crash. Crash detection is implemented by parsing the output of `ps`, and by sending a 0 signal to the process ID. Zombie processes are displayed in `ps` and respond to the 0 signal, so Phusion Passenger thinks the process is still alive even though it has terminated.
  - tjholowaychuk 12y ago
    Fair enough!
  - davidw 12y ago
    > Crash detection is implemented by parsing the output of `ps` Would there be anything to be gained by actually rooting around in /proc yourselves, rather than relying on ps?
    - FooBarWidget 12y ago
      Ps is implemented by parsing /proc, so the results you get are the same. Having said that, /proc/xxx/status does contain a flag that tells you whether a process is a zombie or not. So on Linux, Phusion Passenger parses that file as an extra check. But not all software does this.
digi_owl 12y ago
Sysv heavyweight?
- _delirium 12y ago
  It's got a reasonable amount of stuff in it. The Debian version of sysvinit (v2.88) is about 10,000 lines of C code. Probably not a huge deal, but it does more than just reaping zombie processes, if that's really all you want.
  - pdw 12y ago
    The actual init command is only about 3000 lines, the rest is for auxiliary programs that you probably wouldn't need for Docker. And 10k lines is still tiny compared to Python :)
- vezzy-fnord 12y ago
  Compared to solutions like runit, it is.
vezzy-fnord 12y ago
This is nice and all, but the way the authors wrote this article gives off this feeling of hubris that they've uncovered some critical yet technically obscure problem. They haven't. This is Unix programming 101 right here. Even if the Docker community has been generally ignorant of this, the problem is not any less glaring or basic. I'm not sure if my_init is really an "init system", though. I'd call it an "init daemon", since it handles the bare minimum of waiting on its children. A system would imply greater complexity.
- FooBarWidget 12y ago
  Author of the article here. Of course it's a basic Unix programming 101... from our points of view! You and I think it's basic because we're so knowledgeable in this area. But this does not hold for the rest of the world. 99 out of 100 people that I've spoken to have absolutely no idea about this problem. I used to link to the Wikipedia article and thought that people would get it. They didn't. People started getting it once I gave 15 minute talks about the problem and describing things with diagrams. Lots of people gave me the suggestion that I should blog about it, and so I did.
  - vezzy-fnord 12y ago
    There is no relative point of view here. This is simply an egregious educational failure where application developers have failed to understand the underpinnings of how their platform handles processes at a high level. Then again, this is how the industry works, I suppose. You rediscover an old paradigm or basic property of your system that has been abstracted out, present it as new, deliver it in a shiny package and reap the fame. Sorry, I was mostly ticked at how you presented this issue in such an alarmist and "problem-reaction-solution" tone. It's just humorous.
dypsilon 12y ago
Is there a ticket in the docker project? I would like to follow the progress on this one. Everything I could find was this ticket: https://github.com/docker/docker/issues/5305 https://github.com/docker/docker/issues/5305
emeraldd 12y ago
Why write your own mini-init instead of using something like supervisord?
- masklinn 12y ago
  Supervisord is not an init and must not be run as pid1, that's explicitly mentioned in its documentation[0]: > It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. There are micro-inits you could use instead though, s6 should be runnable as pid1 for instance. [0] http://supervisord.org/index.html?highlight=init http://supervisord.org/index.html?highlight=init
  - emeraldd 12y ago
    I'm a little confused here. Especially considering that the docker site lists supervisord explicitly with instructions on how to use it .... see: https://docs.docker.com/articles/using_supervisord/ https://docs.docker.com/articles/using_supervisord/
    - masklinn 12y ago
      There's no problem with running supervisord, inside or outside of docker, there's a problem running supervisord as an init (pid1). You can run openrc or systemd or s6 or runit or minit or whatever and use supervisord on top of that, you just shouldn't use supervisord directly as pid1.
- deleted 12y ago
  [deleted]
amalag 12y ago
How does this compare to minit or runit ?
sandGorgon 12y ago
I'm not sure writing yet another pid 1 tool is such a good idea. I have been using Docker in production for a few months now and I use supervisord as pid 1. I am able to hook it to most app servers (python,rails,etc) as well as rsyslogd . logging is a big pain on Docker, and you should think of your logging strategy before the pid 1. How are you rotating your logs and is your pid 1 restarting your app on failure? For us, supervisord does all that. We run syslog and logrotate inside the container (using a volume) and it works fairly well for us. Plus supervisord comes with a pretty kickass tool - supervisorctl - and the conf file is fairly well documented. Optimizing the RAM/CPU usage of supervisors and rsyslogd is not very critical - it doesn't use too much. I actually learned a lot from baseimage - but dont know why they went with runit rather than supervisord. Would really,really,REALLY love to learn to do thus with systemd. EDIT: ahh.. forgot that their script was based on runit. My mistake - but dont think I will change over from supervisord anytime soon.
- detaro 12y ago
  Supervisord docs say explicitly that it is not meant be used as PID 1. Does it still perform all the necessary functions?
  - sandGorgon 12y ago
    yes - we use django, rails, postgres, redis, syslogd and cron. In general it works very well - we use puma for the rails server and gunicorn for python and we are able to control them very well.
- deleted 12y ago
  [deleted]
KaiserPro 12y ago
I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as a buildeng i've seen little evidence of that. We use VMware with throwaway build disks. This means that a Dev can destroy a machine, once rebooted its back to a known default state. the best part is that we have collectd on the vmware host, and the in the guest to collect metrics on CPU, mem and IO. Because we have decent disks, and lots of memory, we don't hit any limitations. For heavy build we have NFS root'd servers with non persistant disks. much faster, and requires almost no thinking about.
- eropple 12y ago
  FWIW, and I say this as a big Docker fan, I had the exact same reaction. Docker, to me, is for immutable stuff where I don't care about the end results of that particular instance of the software. Turning containers into thinner VMs doesn't do much for me.
- gizmo686 12y ago
  This is not talking about writing a new init in the sense of Systemd. All the init program needs to do is spawn the one process that you were going to start anyway, and adopt orphan processes. You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').
- koffiezet 12y ago
  Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility. Once you understand the power of it all, and how to think with Docker (which isn't that obvious) - it's pretty damn good. Not that there are no downsides or things that could be improved, but nothing is perfect. And then: 1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done. 2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times. 3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time. My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.
m00dy 12y ago
>“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.” To the author-> Do you know how many processes a kernel can handle ? root@lisa2:~# cat /proc/sys/kernel/pid_max 32768 fyi.
- FooBarWidget 12y ago
  Murphey's law. Do you know how many values a 32-bit integer can represent? Surely no one ever has to worry about overflows?
- moe 12y ago
  This is actually a flexible limit; # echo 1000000 >/proc/sys/kernel/pid_max # cat /proc/sys/kernel/pid_max 1000000 But why would you run your containers with a ticking timebomb?
- ekimekim 12y ago
  Suppose a service has a timed routine that spawns a child every 10 seconds. Unfortunately, this child has a bug and starts leaking zombies to PID1. Congratulations, you now have approx 9 hours until your entire system stops being able to spawn new processes.
  - m00dy 12y ago
    You could have a nuclear war near the datacenter where you host that broken software & container. I think it has a better probability to exist in a real world.
rubiquity 12y ago
I think needing to run an init system with Docker is a solution to a problem you've created for yourself. From reading this post and other posts on your blog about Docker it sounds like you are `docker start`-ing containers that have been `docker stop`-ped. In my experience, using `docker start` isn't a best practice and I design my containers in a way that I won't need to restart a stopped container. If you use Docker containers as a form of lightweight immutable infrastructure you won't care about zombie processes because you won't be restarting stopped containers. Instead you just `docker run` every time you want to run a new container. The example of a corrupt file is moot because your containers shouldn't be writing to the file system with the intention of that file being around for long. This also applies to databases, which I'm not fond of running in containers at all (setting aside highly distributed databases, perhaps).
- ekimekim 12y ago
  The problem isn't when your main docker process is dying (in this case I believe dockerd itself reaps the process), the problem is when your service spawns child processes as part of normal operation. As an example, suppose we have a simple web server that executes CGI scripts written in bash (I know this is kinda contrived but it illustrates my point). I do a request to the server, it runs a bash script. The script runs grep. While grep is executing, the web server decides bash has taken too long and SIGKILLs it. Then grep is adopted by your PID1, in this case your webserver. But the webserver doesn't know its there, and so never calls wait(). You now have a zombie process that will last until you stop the container.
Alupis 12y ago
The terminology used to describe children processes can often seem comical. I recall my systems programming classes and the professor would shout "The parent must kill all the children!" -- was good for a few chuckles. More seriously, the issues raised in this article are real, and I do believe the author has done a fine job breaking down the UNIX process hierarchy and defined responsibilities. I'd say it's a shame as this is yet one more important stability/scalability/security/performance item the Docker team has not thought of (or maybe their busy writing more eco-system apps), but I do believe this is a container problem at large seeing that the official App Container Spec has not addresses it. For those criticizing the need for "yet another pid 1", well, there doesn't seem to be any way around it that can guarantee the same outcome. Yes, it's understandable having reservations about Python being the tool of choice for such an init system, and perhaps C would be a better choice allowing more fine-grained control and guarantees. My guess is the article author was most familiar with Python and wanted to have a POC for the write-up, which is satisfactory.
koffiezet 12y ago
as I posted on reddit: A simpler and better solution for this problem is: do not let apps run in background (double-fork/detach) in containers. Most applications that background by default can override this with a command-line flag. And certainly don't run programs that execute other programs that background. Keep it simple, and try to limit yourself to 1 process per container. Sure there might be situations where this cannot be avoided, but in 99% of the situations - this is not a problem. Yes you have to be aware of this potential problem, but I am currently running tons of containers, of which only a few run supervisord (which handles reaping too) to run multiple processes within one container because it was the sane thing to do. Stale zombie processes on my docker hosts? None. These precautions in my opinion are only required for badly behaving software. And yes - there is software like that.
jrelsasser 12y ago
FWIW I have written a very, very small /sbin/init replacement for docker containers. It runs one script on startup and another on shutdown, supplied by the container author: https://github.com/jre/dinit https://github.com/jre/dinit
shykes 12y ago
Hi, Docker author. Just wanted to say that we are aware of the problem, understand the underlying mechanics, and plan to fix it. In the meantime application-level fixes like this can definitely help. One slight difficulty is that some applications do want to be pid1. While 99% of the ecosystem wants the problem of being pid1 to be handled natively by Docker, the remaining 1% (people who write init systems, or platforms tied to a particular init system) emphatically wants the opposite. So we need to make sure both modes of operations are available, and deal with a dilemma: do we change the default behavior, potentially breaking images expecting to be run as pid1? Or do we keep a default which breaks 99% of applications, requiring an explicit setting to opt into the correct behavior? Anybody with opinions on the matter is welcome to voice them on #docker-dev / freenode, or the dev mailing list, we'd love to hear from you.
- edwintorok 12y ago
  As long as you provide a choice on what to run as PID1, and that init system immediately/automatically runs the ENTRYPOINT / CMD / run command I think it is fine to change the behaviour to the correct one. Please don't make the mistake to hardcode PID1, and worst of all not to systemd ... as systemd causes enough trouble in docker containers already (try debian host, fedora:20 docker and upgrade if you can).
  - shykes 12y ago
    Don't worry, we believe in small composable tools, for example we are planning to break up Docker into more discrete binaries. So we're not about to bundle systemd into every Docker installation ;)
- FooBarWidget 12y ago
  Having a config option is a good idea. I vote for changing the default behavior because Docker has so much growth potential left. Leaving the default as-is is problematic for the majority of people, both current and future. I think that breaking it is acceptable because a lot of the people who use Docker in production -- especially those that are expecting to be PID 1 -- are early adopters and will forgive you for the slight breakage.

119 points by ohcoder 12y ago

Surely writing a minimal init in C (or Go if you like) is better than using Python... it is not a lot of code, and frankly I would trust it not to die and kill your container more. Plus Python has a chunk of dependencies you don't need in a minimal container.

FooBarWidget 12y ago
Lots of system tools are written in Python nowadays. If you use a Debian, Ubuntu or CentOS base image then there's a good chance that Python is preinstalled because some system tool depends on it. It was also much easier to write the tool in Python than in C. Writing it in C would have taken much longer, and at the end of the day we wanted to get work done instead of worrying about a few MBs, which only take 1 second to download and which cost $0.001 of disk space.
- mborch 12y ago
  It's trivial to write this program in C and it is a waste of memory to load up 50 MB of Python just to accomplish this simple task.
  - FooBarWidget 12y ago
    Actually, the Python process eats less than 5 MB of RAM. Most other stuff on the system eats more RAM. Writing my_init took much more time than you think, and I do not regret choosing Python. My_init not only handles process reaping, but also environment variable handling and bunch of other stuff. Writing text parsers in C is a huge pain (libc has no regexps that are as easy to use as Python). I do not oppose rewriting it in C, but given limited manpower and the tiny benefits it's not high on my todo list either. If you are worried about resource usage, then I would very much welcome contributions for rewriting it in C.
    - mborch 12y ago
      Fair enough, but for the record it's at least 10 MB (try starting up Python and import your list of modules). The environment variable handling and other stuff could perhaps be handled in a separate bash-script (or Python), such that the scope of the actual reaper is sufficiently small (to use a lower-level language).
      - kstenerud 12y ago
        5-10MB is a big nothing, unless you're talking about running hundreds of images that consume 20-30mb each. in which case you would have some custom image structure to minimize everything anyway. But then you'd want to consider the infrastructure development cost vs just buying more iron. There comes a point where you have to take a look at the diminishing returns and say "That's good enough."
    - pdw 12y ago
      libc does have regexps. http://pubs.opengroup.org/onlinepubs/009695399/functions/regcomp.html http://pubs.opengroup.org/onlinepubs/009695399/functions/reg...
    - edwintorok 12y ago
      Why not use already existing init? runit seems small enough: http://smarden.org/runit/ http://smarden.org/runit/ Or if you want small have a look here for minimal PID 1 "So how should init be done right?" http://ewontfix.com/14/ http://ewontfix.com/14/
      - FooBarWidget 12y ago
        Baseimage-docker does use Runit. my_init calls Runit.
        edwintorok 12y ago
        Why not use http://smarden.org/runit/runit-init.8.html http://smarden.org/runit/runit-init.8.html and run my_init from runit?
- falcolas 12y ago
  Writing it in Python also adds a lot of dependencies and bloat to the docker image which don't really need to be there. Particularly when there are simple init scripts like [1] which could be easily adapted to fill both the reaping and restart problem. [1] http://git.suckless.org/sinit/tree/sinit.c http://git.suckless.org/sinit/tree/sinit.c
- justincormack 12y ago
  It really does not need much code in C, here is a simple init https://github.com/arachsys/init/blob/master/init.c https://github.com/arachsys/init/blob/master/init.c for example. In particular you want to avoid memory allocation so it cannot fail even in low memory situations, which you cannot guarantee with Python. That means you might die reaping children with low memory which would kill your container.
dnr 12y ago
Indeed: I solved this minor issue with a 23 line C program that compiles to a 8766 byte binary. It's really not a big deal. (Mine doesn't pass on signals, but that's not important for us.)

tjholowaychuk 12y ago

Isn't this effectively a non-issue? It's literally just a tiny entry in a table that is held no? Or does it just cause Docker to behave oddly?

FooBarWidget 12y ago
There are two issues. First is Murphey's law. 1 zombie process is probably harmless, but if the amount grows then you'll fill up kernel limits and the container cannot create further processes. Second, zombie processes that don't go away can also interfere with software that check for the existence of processes. For example, [the Phusion Passenger application server](https://www.phusionpassenger.com/ https://www.phusionpassenger.com/) manages processes. It restarts processes when they crash. Crash detection is implemented by parsing the output of `ps`, and by sending a 0 signal to the process ID. Zombie processes are displayed in `ps` and respond to the 0 signal, so Phusion Passenger thinks the process is still alive even though it has terminated.
- tjholowaychuk 12y ago
  Fair enough!
- davidw 12y ago
  > Crash detection is implemented by parsing the output of `ps` Would there be anything to be gained by actually rooting around in /proc yourselves, rather than relying on ps?
  - FooBarWidget 12y ago
    Ps is implemented by parsing /proc, so the results you get are the same. Having said that, /proc/xxx/status does contain a flag that tells you whether a process is a zombie or not. So on Linux, Phusion Passenger parses that file as an extra check. But not all software does this.

digi_owl 12y ago

Sysv heavyweight?

_delirium 12y ago
It's got a reasonable amount of stuff in it. The Debian version of sysvinit (v2.88) is about 10,000 lines of C code. Probably not a huge deal, but it does more than just reaping zombie processes, if that's really all you want.
- pdw 12y ago
  The actual init command is only about 3000 lines, the rest is for auxiliary programs that you probably wouldn't need for Docker. And 10k lines is still tiny compared to Python :)
vezzy-fnord 12y ago
Compared to solutions like runit, it is.

vezzy-fnord 12y ago

This is nice and all, but the way the authors wrote this article gives off this feeling of hubris that they've uncovered some critical yet technically obscure problem. They haven't. This is Unix programming 101 right here. Even if the Docker community has been generally ignorant of this, the problem is not any less glaring or basic. I'm not sure if my_init is really an "init system", though. I'd call it an "init daemon", since it handles the bare minimum of waiting on its children. A system would imply greater complexity.

FooBarWidget 12y ago
Author of the article here. Of course it's a basic Unix programming 101... from our points of view! You and I think it's basic because we're so knowledgeable in this area. But this does not hold for the rest of the world. 99 out of 100 people that I've spoken to have absolutely no idea about this problem. I used to link to the Wikipedia article and thought that people would get it. They didn't. People started getting it once I gave 15 minute talks about the problem and describing things with diagrams. Lots of people gave me the suggestion that I should blog about it, and so I did.
- vezzy-fnord 12y ago
  There is no relative point of view here. This is simply an egregious educational failure where application developers have failed to understand the underpinnings of how their platform handles processes at a high level. Then again, this is how the industry works, I suppose. You rediscover an old paradigm or basic property of your system that has been abstracted out, present it as new, deliver it in a shiny package and reap the fame. Sorry, I was mostly ticked at how you presented this issue in such an alarmist and "problem-reaction-solution" tone. It's just humorous.

dypsilon 12y ago

Is there a ticket in the docker project? I would like to follow the progress on this one. Everything I could find was this ticket: https://github.com/docker/docker/issues/5305 https://github.com/docker/docker/issues/5305

emeraldd 12y ago

Why write your own mini-init instead of using something like supervisord?

masklinn 12y ago
Supervisord is not an init and must not be run as pid1, that's explicitly mentioned in its documentation[0]: > It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. There are micro-inits you could use instead though, s6 should be runnable as pid1 for instance. [0] http://supervisord.org/index.html?highlight=init http://supervisord.org/index.html?highlight=init
- emeraldd 12y ago
  I'm a little confused here. Especially considering that the docker site lists supervisord explicitly with instructions on how to use it .... see: https://docs.docker.com/articles/using_supervisord/ https://docs.docker.com/articles/using_supervisord/
  - masklinn 12y ago
    There's no problem with running supervisord, inside or outside of docker, there's a problem running supervisord as an init (pid1). You can run openrc or systemd or s6 or runit or minit or whatever and use supervisord on top of that, you just shouldn't use supervisord directly as pid1.
deleted 12y ago
[deleted]

amalag 12y ago

How does this compare to minit or runit ?

sandGorgon 12y ago

I'm not sure writing yet another pid 1 tool is such a good idea. I have been using Docker in production for a few months now and I use supervisord as pid 1. I am able to hook it to most app servers (python,rails,etc) as well as rsyslogd . logging is a big pain on Docker, and you should think of your logging strategy before the pid 1. How are you rotating your logs and is your pid 1 restarting your app on failure? For us, supervisord does all that. We run syslog and logrotate inside the container (using a volume) and it works fairly well for us. Plus supervisord comes with a pretty kickass tool - supervisorctl - and the conf file is fairly well documented. Optimizing the RAM/CPU usage of supervisors and rsyslogd is not very critical - it doesn't use too much. I actually learned a lot from baseimage - but dont know why they went with runit rather than supervisord. Would really,really,REALLY love to learn to do thus with systemd. EDIT: ahh.. forgot that their script was based on runit. My mistake - but dont think I will change over from supervisord anytime soon.

detaro 12y ago
Supervisord docs say explicitly that it is not meant be used as PID 1. Does it still perform all the necessary functions?
- sandGorgon 12y ago
  yes - we use django, rails, postgres, redis, syslogd and cron. In general it works very well - we use puma for the rails server and gunicorn for python and we are able to control them very well.
deleted 12y ago
[deleted]

KaiserPro 12y ago

I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as a buildeng i've seen little evidence of that. We use VMware with throwaway build disks. This means that a Dev can destroy a machine, once rebooted its back to a known default state. the best part is that we have collectd on the vmware host, and the in the guest to collect metrics on CPU, mem and IO. Because we have decent disks, and lots of memory, we don't hit any limitations. For heavy build we have NFS root'd servers with non persistant disks. much faster, and requires almost no thinking about.

eropple 12y ago
FWIW, and I say this as a big Docker fan, I had the exact same reaction. Docker, to me, is for immutable stuff where I don't care about the end results of that particular instance of the software. Turning containers into thinner VMs doesn't do much for me.
gizmo686 12y ago
This is not talking about writing a new init in the sense of Systemd. All the init program needs to do is spawn the one process that you were going to start anyway, and adopt orphan processes. You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').
koffiezet 12y ago
Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility. Once you understand the power of it all, and how to think with Docker (which isn't that obvious) - it's pretty damn good. Not that there are no downsides or things that could be improved, but nothing is perfect. And then: 1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done. 2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times. 3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time. My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.

m00dy 12y ago

>“As long as a zombie is not removed from the system via a wait, it will consume a slot in the kernel process table, and if this table fills, it will not be possible to create further processes.” To the author-> Do you know how many processes a kernel can handle ? root@lisa2:~# cat /proc/sys/kernel/pid_max 32768 fyi.

FooBarWidget 12y ago
Murphey's law. Do you know how many values a 32-bit integer can represent? Surely no one ever has to worry about overflows?
moe 12y ago
This is actually a flexible limit; # echo 1000000 >/proc/sys/kernel/pid_max # cat /proc/sys/kernel/pid_max 1000000 But why would you run your containers with a ticking timebomb?
ekimekim 12y ago
Suppose a service has a timed routine that spawns a child every 10 seconds. Unfortunately, this child has a bug and starts leaking zombies to PID1. Congratulations, you now have approx 9 hours until your entire system stops being able to spawn new processes.
- m00dy 12y ago
  You could have a nuclear war near the datacenter where you host that broken software & container. I think it has a better probability to exist in a real world.

rubiquity 12y ago

I think needing to run an init system with Docker is a solution to a problem you've created for yourself. From reading this post and other posts on your blog about Docker it sounds like you are `docker start`-ing containers that have been `docker stop`-ped. In my experience, using `docker start` isn't a best practice and I design my containers in a way that I won't need to restart a stopped container. If you use Docker containers as a form of lightweight immutable infrastructure you won't care about zombie processes because you won't be restarting stopped containers. Instead you just `docker run` every time you want to run a new container. The example of a corrupt file is moot because your containers shouldn't be writing to the file system with the intention of that file being around for long. This also applies to databases, which I'm not fond of running in containers at all (setting aside highly distributed databases, perhaps).

ekimekim 12y ago
The problem isn't when your main docker process is dying (in this case I believe dockerd itself reaps the process), the problem is when your service spawns child processes as part of normal operation. As an example, suppose we have a simple web server that executes CGI scripts written in bash (I know this is kinda contrived but it illustrates my point). I do a request to the server, it runs a bash script. The script runs grep. While grep is executing, the web server decides bash has taken too long and SIGKILLs it. Then grep is adopted by your PID1, in this case your webserver. But the webserver doesn't know its there, and so never calls wait(). You now have a zombie process that will last until you stop the container.

Alupis 12y ago

The terminology used to describe children processes can often seem comical. I recall my systems programming classes and the professor would shout "The parent must kill all the children!" -- was good for a few chuckles. More seriously, the issues raised in this article are real, and I do believe the author has done a fine job breaking down the UNIX process hierarchy and defined responsibilities. I'd say it's a shame as this is yet one more important stability/scalability/security/performance item the Docker team has not thought of (or maybe their busy writing more eco-system apps), but I do believe this is a container problem at large seeing that the official App Container Spec has not addresses it. For those criticizing the need for "yet another pid 1", well, there doesn't seem to be any way around it that can guarantee the same outcome. Yes, it's understandable having reservations about Python being the tool of choice for such an init system, and perhaps C would be a better choice allowing more fine-grained control and guarantees. My guess is the article author was most familiar with Python and wanted to have a POC for the write-up, which is satisfactory.

koffiezet 12y ago

as I posted on reddit: A simpler and better solution for this problem is: do not let apps run in background (double-fork/detach) in containers. Most applications that background by default can override this with a command-line flag. And certainly don't run programs that execute other programs that background. Keep it simple, and try to limit yourself to 1 process per container. Sure there might be situations where this cannot be avoided, but in 99% of the situations - this is not a problem. Yes you have to be aware of this potential problem, but I am currently running tons of containers, of which only a few run supervisord (which handles reaping too) to run multiple processes within one container because it was the sane thing to do. Stale zombie processes on my docker hosts? None. These precautions in my opinion are only required for badly behaving software. And yes - there is software like that.

jrelsasser 12y ago

FWIW I have written a very, very small /sbin/init replacement for docker containers. It runs one script on startup and another on shutdown, supplied by the container author: https://github.com/jre/dinit https://github.com/jre/dinit

shykes 12y ago

Hi, Docker author. Just wanted to say that we are aware of the problem, understand the underlying mechanics, and plan to fix it. In the meantime application-level fixes like this can definitely help. One slight difficulty is that some applications do want to be pid1. While 99% of the ecosystem wants the problem of being pid1 to be handled natively by Docker, the remaining 1% (people who write init systems, or platforms tied to a particular init system) emphatically wants the opposite. So we need to make sure both modes of operations are available, and deal with a dilemma: do we change the default behavior, potentially breaking images expecting to be run as pid1? Or do we keep a default which breaks 99% of applications, requiring an explicit setting to opt into the correct behavior? Anybody with opinions on the matter is welcome to voice them on #docker-dev / freenode, or the dev mailing list, we'd love to hear from you.

edwintorok 12y ago
As long as you provide a choice on what to run as PID1, and that init system immediately/automatically runs the ENTRYPOINT / CMD / run command I think it is fine to change the behaviour to the correct one. Please don't make the mistake to hardcode PID1, and worst of all not to systemd ... as systemd causes enough trouble in docker containers already (try debian host, fedora:20 docker and upgrade if you can).
- shykes 12y ago
  Don't worry, we believe in small composable tools, for example we are planning to break up Docker into more discrete binaries. So we're not about to bundle systemd into every Docker installation ;)
FooBarWidget 12y ago
Having a config option is a good idea. I vote for changing the default behavior because Docker has so much growth potential left. Leaving the default as-is is problematic for the majority of people, both current and future. I think that breaking it is acceptable because a lot of the people who use Docker in production -- especially those that are expecting to be PID 1 -- are early adopters and will forgive you for the slight breakage.