6 ms·

I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker,

I know this will cost me a lot of karma, but surely this is the point that you say, lets just use a hypervisor? yes, I know its much easier to just use docker, but at the point that you have to write a new INIT, doesn't that strike you as time you used something thats not easy anymore? Things like "make sure we don't loose syslog" make me shudder. Yes supposedly there is an argument that docker is faster. However as a buildeng i've seen little evidence of that. We use VMware with throwaway build disks. This means that a Dev can destroy a machine, once rebooted its back to a known default state. the best part is that we have collectd on the vmware host, and the in the guest to collect metrics on CPU, mem and IO. Because we have decent disks, and lots of memory, we don't hit any limitations. For heavy build we have NFS root'd servers with non persistant disks. much faster, and requires almost no thinking about.

eropple 12y ago
FWIW, and I say this as a big Docker fan, I had the exact same reaction. Docker, to me, is for immutable stuff where I don't care about the end results of that particular instance of the software. Turning containers into thinner VMs doesn't do much for me.
gizmo686 12y ago
This is not talking about writing a new init in the sense of Systemd. All the init program needs to do is spawn the one process that you were going to start anyway, and adopt orphan processes. You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').
koffiezet 12y ago
Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility. Once you understand the power of it all, and how to think with Docker (which isn't that obvious) - it's pretty damn good. Not that there are no downsides or things that could be improved, but nothing is perfect. And then: 1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done. 2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times. 3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time. My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.
- shykes 12y ago
  > My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications. You should take a look at https://github.com/docker/machine https://github.com/docker/machine and https://github.com/docker/swarm https://github.com/docker/swarm , they were designed with exactly that goal in mind!
  - KaiserPro 12y ago
    Sadly, it still lags behind even KVM, it can't hot migrate, which means zero downtime maintenance is impossible, along with transparent HA.
- dordoka 12y ago
  Care to share your CI tech stack? Specially the deployment bit automagically updating the balancer. We are internally building that kind of setup at the moment and some insight from experienced people like you would be very helpful. Thanks.
  - koffiezet 12y ago
    CI is Jenkins, which uses the Docker plugin for creating slaves on-the-fly. The plugin has it's share of problems, so I don't use it to build the images itself - I do this using a generalized bash script, which also pushes it to our private repo. For automatically updating the load-balancer, check out docker-gen (https://github.com/jwilder/docker-gen https://github.com/jwilder/docker-gen). I hacked together setup that detects the latest version and points to this-one when a container comes up or goes down. It assumes a lot of things specific to my setup though (versioning of the containers, env vars, ...)
bitwize 12y ago
This is actually why we all need to support systemd. It is already an init system, and a sophisticated one; now that it has integrated container support it's only a matter of time before systemd's pid 1 assumes zombie reaping duties for the host and all container images. People who criticize systemd fail to realize that it provides solutoons for many problems under Linux, and even problems with the Unix model itself, and has widespread support from companies, distributions, and other stakeholders.
- TheSwordsman 12y ago
  And people who blindly push systemd fail to realize that we've alreqady solved these problems. A more complex system doesn't solve it. So instead of a 350 line Python script, you think systemd is better? I disagree.
  - xyzzy_plugh 12y ago
    A more complex system? How is solving this problem universally a bad thing? Is having _one_ way to do things that works everywhere so bad? Having a container manager do things I wouldn't expect, like have issues with zombies or dropping syslog messages, makes me never want to touch it.
    - KaiserPro 12y ago
      My dear chap, one way does not work everywhere. if that were the case we'd all have agreed to just use vim, or emacs, or sublime, or nedit, or nano, or ed......
      - bitwize 12y ago
        Vim has largely won vs. emacs, and gone on to the next bracket against ST. Emacs is out of the tourney. I can't thibk of a single hacker under 30 who isn't a total Lisphead, who wants to even touch emacs.
- KaiserPro 12y ago
teh 12y ago
A hypervisor has a ~150-250MB memory overhead over linux containers + namespaces because it boots a full kernel. There are other differences but if all you want from your system is reproducibility then VM images work fine of course.

eropple 12y ago

FWIW, and I say this as a big Docker fan, I had the exact same reaction. Docker, to me, is for immutable stuff where I don't care about the end results of that particular instance of the software. Turning containers into thinner VMs doesn't do much for me.

gizmo686 12y ago

This is not talking about writing a new init in the sense of Systemd. All the init program needs to do is spawn the one process that you were going to start anyway, and adopt orphan processes. You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').

koffiezet 12y ago

Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility. Once you understand the power of it all, and how to think with Docker (which isn't that obvious) - it's pretty damn good. Not that there are no downsides or things that could be improved, but nothing is perfect. And then: 1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done. 2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times. 3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time. My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.

shykes 12y ago
> My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications. You should take a look at https://github.com/docker/machine https://github.com/docker/machine and https://github.com/docker/swarm https://github.com/docker/swarm , they were designed with exactly that goal in mind!
- KaiserPro 12y ago
  Sadly, it still lags behind even KVM, it can't hot migrate, which means zero downtime maintenance is impossible, along with transparent HA.
dordoka 12y ago
Care to share your CI tech stack? Specially the deployment bit automagically updating the balancer. We are internally building that kind of setup at the moment and some insight from experienced people like you would be very helpful. Thanks.
- koffiezet 12y ago
  CI is Jenkins, which uses the Docker plugin for creating slaves on-the-fly. The plugin has it's share of problems, so I don't use it to build the images itself - I do this using a generalized bash script, which also pushes it to our private repo. For automatically updating the load-balancer, check out docker-gen (https://github.com/jwilder/docker-gen https://github.com/jwilder/docker-gen). I hacked together setup that detects the latest version and points to this-one when a container comes up or goes down. It assumes a lot of things specific to my setup though (versioning of the containers, env vars, ...)

bitwize 12y ago

This is actually why we all need to support systemd. It is already an init system, and a sophisticated one; now that it has integrated container support it's only a matter of time before systemd's pid 1 assumes zombie reaping duties for the host and all container images. People who criticize systemd fail to realize that it provides solutoons for many problems under Linux, and even problems with the Unix model itself, and has widespread support from companies, distributions, and other stakeholders.

TheSwordsman 12y ago
And people who blindly push systemd fail to realize that we've alreqady solved these problems. A more complex system doesn't solve it. So instead of a 350 line Python script, you think systemd is better? I disagree.
- xyzzy_plugh 12y ago
  A more complex system? How is solving this problem universally a bad thing? Is having _one_ way to do things that works everywhere so bad? Having a container manager do things I wouldn't expect, like have issues with zombies or dropping syslog messages, makes me never want to touch it.
  - KaiserPro 12y ago
    My dear chap, one way does not work everywhere. if that were the case we'd all have agreed to just use vim, or emacs, or sublime, or nedit, or nano, or ed......
    - bitwize 12y ago
      Vim has largely won vs. emacs, and gone on to the next bracket against ST. Emacs is out of the tourney. I can't thibk of a single hacker under 30 who isn't a total Lisphead, who wants to even touch emacs.
KaiserPro 12y ago

teh 12y ago

A hypervisor has a ~150-250MB memory overhead over linux containers + namespaces because it boots a full kernel. There are other differences but if all you want from your system is reproducibility then VM images work fine of course.