7 ms·

Author of the article here; ping me if you have any questions or comments :-).

patrickaljord 12y ago
Just a little comment, it says Docker is an abstraction layer on top of LXC. Since version 0.9, Docker uses their own libcontainer go package to communicate with the kernel directly instead of using LXC. It is still possible to use LXC though, but it's not used by default. http://blog.docker.io/2014/03/docker-0-9-introducing-execution-drivers-and-libcontainer/ http://blog.docker.io/2014/03/docker-0-9-introducing-executi...
- marceldegraaf 12y ago
  Thanks for the heads-up, I've removed the reference to LXC.
- tokenizerrr 12y ago
  Is the package name[1] for that still lxc-docker? [1]: http://docs.docker.io/installation/ubuntulinux/ http://docs.docker.io/installation/ubuntulinux/
  - bacongobbler 12y ago
    yes, it's still lxc-docker.
    - nickstinemates 12y ago
      It's an unfortunate situation we're working to rectify.
bacongobbler 12y ago
A small enhancement to your HOST_IP grepping, the host's public and private IP is available in /etc/environment. docker0 may or may not be present before a docker run as well due to socket activation, so your results may vary with a brand new CoreOS VM where docker has not started yet. For example, we are resolving this in Deis with https://github.com/deis/deis/pull/851 https://github.com/deis/deis/pull/851
- marceldegraaf 12y ago
  Ha, thanks! I actually got the grepping magic from Deis as well, but this is a much more elegant solution. By the way, Deis is looking really good now that it's CoreOS based :-).
- robszumski 12y ago
  To expand on this a bit, you can source a file to expose variables to your unit. On most providers, CoreOS populates /etc/environment with the public and private IP. You can then use those as env vars in your unit: [Service] EnvironmentFile=/etc/environment You can then use $COREOS_PUBLIC_IPV4 anywhere in that unit file. Makes service registration a breeze.
  - marceldegraaf 12y ago
    Hi Rob, thanks for your expansion on it :-). I see Deis does that in a similar fashion. I'll update the article with this approach, because it's clearly the best way to solve the "host IP issue".
krschultz 12y ago
I am wondering why you have a 1:1 mapping between Docker containers and processes? I have not used Docker but my impression of it was a good way to group a few inter-related processes together while separating them from unrelated processes. Our thought was basically 1 container per role/service in our platform (app servers, workers, DB, logging, etc). If you have 1 process per container, isn't it just a bit stronger form of sandboxing a process?
- marceldegraaf 12y ago
  That's a fair point. In this case I was especially interested in running multiple instances of the same application on a CoreOS mini-cluster, demonstrating fleet's capabilities of scaling/distributing the containers over the cluster. However, in a real life application I think I would still keep the containers as "single-process" as possible; in our case that would mean: a single Rails application or a group of Resque workers. It would make the containers quick to boot and would increase security by sandboxing all the separate parts of a system in their own containers. I do think this is up to personal preferences, though :-).
- justincormack 12y ago
  The process is the unit in which things fail in an OS, so a collection of processes needs to do its own failure handling. Part of the point of these types of system is to do things like failure handling, rather than everything having to do its own.
- danudey 12y ago
  You can do it in several different ways, and it really depends on what you're trying to accomplish. If you were trying to build wordpress-as-a-service, for example, you might create a Docker instance which includes nginx, php-fpm, memcached, and maybe even mysql (which makes it easy to make snapshots of the entire state of the system). That said, fleet has a really interesting way of making interconnected systems, so you can create a memcached instance, an nginx instance, and a php-fpm-running-wordpress instance, and you can manage them separately. This makes it easy to upgrade memcached without touching nginx or php-fpm, or to change the nginx config without having to affect the running application server (and risk flushing the APC cache), etc. It also allows you to make everything pluggable in your design. You might want to add varnish in front of your nginx instance, for example, but sometimes you might not. That sort of flexibility is really useful in some circumstances. In our infrastructure at work, the design I would use would probably be similar to this; we have front-end services (nginx), internal services that they proxy to, internal services that those services access, asynchronous processing services, dispatch services, activemq, redis, memcached, mysql, etc. Having the flexibility to add more of any one of those things without having to scale any of the rest is really important. For other circumstances, it makes a lot more sense to bundle everything into a faux-VM. Run supervisord to launch ssh, nginx, php-fpm, memcached, and whatever else needs running. Share volumes to local filesystems so that configs, local changes, etc. can be taken care of, and you're good to go. So in the end, the real question is what do you want out of your containers? Flexibility or simplicity?
cridenour 12y ago
Been looking for an article that puts all these pieces together as I haven't had time to experiment with it myself. Thank you!
- marceldegraaf 12y ago
  You're welcome! I'm now looking into log aggregation and centralisation with Logstash and logstash-forwarder (previously Lumberjack). So expect a new article on that in the near future.
dvanduzer 12y ago
Have you done any stress testing yet? I've run into some contention problems with more than a couple hundred containers per node. I imagine most other deployments haven't required that kind of density so far...
- robszumski 12y ago
  What types of issues are you seeing? From time to time it's easy to get hundreds (or thousands) of containers on a machine. I know docker is a bit slow to start up with a lot of containers, but because of the socket activation that shouldn't be an issue at all if you're starting containers via systemd or fleet.
  - dvanduzer 12y ago
    I gave Blake the whole rundown (i.e. laundry list of whines) at GopherCon, and it sounds like a lot of the issues I ran into are fixed in more recent images. I was attempting to inject 1000 containers running a small node.js app into 3 168GB AWS instances via fleet 0.1.3 (I think). Part of my problem may have been that I was sharing some config data between app instances via Redis, when I could've just used etcd for that.
- marceldegraaf 12y ago
  Nope, I haven't. I'm just getting started with all this, so I've got a lot to explore and learn... :-)

Just a little comment, it says Docker is an abstraction layer on top of LXC. Since version 0.9, Docker uses their own libcontainer go package to communicate with the kernel directly instead of using LXC. It is still possible to use LXC though, but it's not used by default. http://blog.docker.io/2014/03/docker-0-9-introducing-execution-drivers-and-libcontainer/ http://blog.docker.io/2014/03/docker-0-9-introducing-executi...

marceldegraaf 12y ago
Thanks for the heads-up, I've removed the reference to LXC.
tokenizerrr 12y ago
Is the package name[1] for that still lxc-docker? [1]: http://docs.docker.io/installation/ubuntulinux/ http://docs.docker.io/installation/ubuntulinux/
- bacongobbler 12y ago
  yes, it's still lxc-docker.
  - nickstinemates 12y ago
    It's an unfortunate situation we're working to rectify.

bacongobbler 12y ago

A small enhancement to your HOST_IP grepping, the host's public and private IP is available in /etc/environment. docker0 may or may not be present before a docker run as well due to socket activation, so your results may vary with a brand new CoreOS VM where docker has not started yet. For example, we are resolving this in Deis with https://github.com/deis/deis/pull/851 https://github.com/deis/deis/pull/851

marceldegraaf 12y ago
Ha, thanks! I actually got the grepping magic from Deis as well, but this is a much more elegant solution. By the way, Deis is looking really good now that it's CoreOS based :-).
robszumski 12y ago
To expand on this a bit, you can source a file to expose variables to your unit. On most providers, CoreOS populates /etc/environment with the public and private IP. You can then use those as env vars in your unit: [Service] EnvironmentFile=/etc/environment You can then use $COREOS_PUBLIC_IPV4 anywhere in that unit file. Makes service registration a breeze.
- marceldegraaf 12y ago
  Hi Rob, thanks for your expansion on it :-). I see Deis does that in a similar fashion. I'll update the article with this approach, because it's clearly the best way to solve the "host IP issue".

krschultz 12y ago

I am wondering why you have a 1:1 mapping between Docker containers and processes? I have not used Docker but my impression of it was a good way to group a few inter-related processes together while separating them from unrelated processes. Our thought was basically 1 container per role/service in our platform (app servers, workers, DB, logging, etc). If you have 1 process per container, isn't it just a bit stronger form of sandboxing a process?

marceldegraaf 12y ago
That's a fair point. In this case I was especially interested in running multiple instances of the same application on a CoreOS mini-cluster, demonstrating fleet's capabilities of scaling/distributing the containers over the cluster. However, in a real life application I think I would still keep the containers as "single-process" as possible; in our case that would mean: a single Rails application or a group of Resque workers. It would make the containers quick to boot and would increase security by sandboxing all the separate parts of a system in their own containers. I do think this is up to personal preferences, though :-).
justincormack 12y ago
The process is the unit in which things fail in an OS, so a collection of processes needs to do its own failure handling. Part of the point of these types of system is to do things like failure handling, rather than everything having to do its own.
danudey 12y ago
You can do it in several different ways, and it really depends on what you're trying to accomplish. If you were trying to build wordpress-as-a-service, for example, you might create a Docker instance which includes nginx, php-fpm, memcached, and maybe even mysql (which makes it easy to make snapshots of the entire state of the system). That said, fleet has a really interesting way of making interconnected systems, so you can create a memcached instance, an nginx instance, and a php-fpm-running-wordpress instance, and you can manage them separately. This makes it easy to upgrade memcached without touching nginx or php-fpm, or to change the nginx config without having to affect the running application server (and risk flushing the APC cache), etc. It also allows you to make everything pluggable in your design. You might want to add varnish in front of your nginx instance, for example, but sometimes you might not. That sort of flexibility is really useful in some circumstances. In our infrastructure at work, the design I would use would probably be similar to this; we have front-end services (nginx), internal services that they proxy to, internal services that those services access, asynchronous processing services, dispatch services, activemq, redis, memcached, mysql, etc. Having the flexibility to add more of any one of those things without having to scale any of the rest is really important. For other circumstances, it makes a lot more sense to bundle everything into a faux-VM. Run supervisord to launch ssh, nginx, php-fpm, memcached, and whatever else needs running. Share volumes to local filesystems so that configs, local changes, etc. can be taken care of, and you're good to go. So in the end, the real question is what do you want out of your containers? Flexibility or simplicity?

cridenour 12y ago

Been looking for an article that puts all these pieces together as I haven't had time to experiment with it myself. Thank you!

marceldegraaf 12y ago
You're welcome! I'm now looking into log aggregation and centralisation with Logstash and logstash-forwarder (previously Lumberjack). So expect a new article on that in the near future.

dvanduzer 12y ago

Have you done any stress testing yet? I've run into some contention problems with more than a couple hundred containers per node. I imagine most other deployments haven't required that kind of density so far...

robszumski 12y ago
What types of issues are you seeing? From time to time it's easy to get hundreds (or thousands) of containers on a machine. I know docker is a bit slow to start up with a lot of containers, but because of the socket activation that shouldn't be an issue at all if you're starting containers via systemd or fleet.
- dvanduzer 12y ago
  I gave Blake the whole rundown (i.e. laundry list of whines) at GopherCon, and it sounds like a lot of the issues I ran into are fixed in more recent images. I was attempting to inject 1000 containers running a small node.js app into 3 168GB AWS instances via fleet 0.1.3 (I think). Part of my problem may have been that I was sharing some config data between app instances via Redis, when I could've just used etcd for that.
marceldegraaf 12y ago
Nope, I haven't. I'm just getting started with all this, so I've got a lot to explore and learn... :-)