7 ms·

Docker containers should not run an SSH server

125 points by davidkellis 12y ago

contingencies 12y ago
Hrrm. I want to agree, but as far as containers go outside of purely docker, I just don't buy these arguments. It makes perfect sense to use industry standard, encrypted communications with proven cryptography when creating vast numbers of systems. Yes, you don't have to. No, that doesn't mean it's a bad idea. I'm not sure why things would be any different with docker, and after reading the article I'm unconvinced. If you want to be locked in to docker's APIs, so be it. If you want to be free of them and integrate in other ways, such as proven, portable, secure methods like SSH, that's fine too. If a management task as simple as deploying a key is so hard in docker (couldn't you just bind-mount a read only .ssh dir?), maybe you should consider alternative methods of container instantiation. I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite.
- ldlework 12y ago
  I don't think I understand your point. From the Docker Host itself, if you need to manage the state of a container, the intuition is that you need to go into the container (with SSH) in order to do so. But by externalizing your state, you can manage it without the need to enter the container. Assuming your Docker Host is secure, this doesn't make anything less secure just because you're no longer abusing SSHd in order to manage your application's state. In the case you need to gdb, or strace the process, you can do that from the Docker Host with nsenter. Assuming your Docker Host is secure, you no longer need to abuse SSHd to carry out a debugging task that has nothing to do with needing a secure shell. Neither of these use-cases have anything to do with the security of SSH. In the case that you need to do these things from a remote host, the prescribed answer is indeed SSHd to access the Docker Host, at which point you switch to the previously suggested methods for managing state. "I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite." Because now you only have to worry about one security layer instead of N security layers for each container you run. The security layer is now actually coupled to the act of granting access to the host, its intended purpose vs granting access to a container so you can manage its state or debug it or whatever. As far as being locked into Docker's APIs, I totally miss the aim of this remark. Volumes are just paths on the filesystem. If you're talking about the interoperability of standard tools to manage your state, I don't think they will have problems in this case.
  - contingencies 12y ago
    the prescribed answer is indeed SSHd to access the Docker Host, at which point you switch to the previously suggested methods for managing state. [...] As far as being locked into Docker's APIs, I totally miss the aim of this remark. Yes, you missed the point. Please read the other response to comprehend the difference.
- vidarh 12y ago
  > I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite. If you are concerned about granting access to the host, you should be concerned about granting access to a Docker container, especially as root, until it's far more battle tested. And if you are concerned about this, there's a simple solution: Group your Docker containers in KVM vm's. If you are not concerned about this, you can easily do as he suggested, and force "nsenter" on login. Or you can do both: Group your containers in a KVM vm and force ssh into the KVM to use nsenter into the appropriate containers. Part of the point is that it makes little sense to add to the complexity and attack surface to add a process monitor per container (to run both sshd and the app) and an sshd per container, even if what you add is something that provides "industry standard, encrypted communications with proven cryptography" if it is not actually needed inside the containers. You're also not getting "locked into" Dockers API much if at all. What is proposing depends on two things: A way of executing a command that can access a certain part of the containers filesystem, and a way of entering the appropriate namespaces. Both are easily abstracted behind tiny (1-2 line) scripts where you can replace "docker run" and "nsenter" with ssh if your needs change. And one (nsenter) is entirely independent of Docker - it will work with e.g. lxc and anything else that makes use of cgroups as well.
  - contingencies 12y ago
    If you are concerned about granting access to the host... Everyone should be. There's a simple solution: Group your Docker containers in KVM vm's. I consider that an ugly hack that should be avoided at all costs. Why? It's not viable without up-front automation investment, ongoing maintenance overheads, and additional latency and inefficiency, for starters. It's also probably less portable, particularly to embedded systems. it makes little sense to add to the complexity and attack surface to add a process monitor per container and an sshd per container Process monitor? What are you talking about? At least with LXC, each container has PID 1 (master application process, init system, whatever) and can be stopped/started easily with lxc-stop -n nameofcontainer. As for sshd, my point was that it can make sense because it's secure, remote-accessible, proven. The article premise was that it's a bad idea, but none of its arguments hold, to my mind. Both are easily abstracted behind tiny (1-2 line) scripts... That aren't remotely accessible, without breaking the abstraction and creating some shared access scenario on the host. The entire point of an abstraction is to be clean. It's not clean if you have to adopt hacky workarounds and dump all of your existing tools in order to play with it.
CraigJPerry 12y ago
I don't get it. What's wrong with putting my SSH public key in the image? Tbh I'd pay more attention to host key regeneration! Easily overlooked. Until there's better investigation tooling available, sshd seems a fairly sensible approach. MAC / SELinux prevents many other poke-inside techniques on enterprise platforms
- waffle_ss 12y ago
  I don't think it's SSHd in particular that he has a problem with. Docker maintainers do not seem to like the idea of people treating Docker containers as lightweight VMs. They seem to want images to be restricted to running as few processes as possible - ideally one. I think they view it as too hard to scale once you start going down the path of running multiple services in an image - I've seen the "pets vs cattle"[1] analogy used for explaining why. There is an image by Phusion called baseimage-docker[2] which adds SSHd, init, syslog, and cron in an attempt to make Docker containers more like lightweight VMs. But in the #docker channel, I've seen people have issues with it. For example, one person had some /etc/init.d scripts that wouldn't start up (other ones started up fine). Turns out that one of the signals that init was waiting on to start that script was never getting sent (I think it was networking coming online?), and that was just a side effect of how Docker works that couldn't easily be fixed. The Docker maintainers in the channel discouraged using this image for these reasons. [1]: https://groups.google.com/forum/#!msg/docker-user/pNaBYJkmnAA/TsA3b8u5kf0J https://groups.google.com/forum/#!msg/docker-user/pNaBYJkmnA... [2]: http://phusion.github.io/baseimage-docker/ http://phusion.github.io/baseimage-docker/
  - opendais 12y ago
    Yes but I bet the majority of people using Docker don't need/want to scale the majority of their projects to 100+ servers which is what the Docker maintainers focus on. It is a good/valid focus for them. However, it is not for everyone.
    - ldlework 12y ago
      I don't scale out to a hundred servers and I still use Docker with separate processes because scaling is not the only advantage to having stateless single-purpose containers.
opendais 12y ago
I think the core flaw with the "no SSHD in Docker, ever" reasoning is it makes a number of assumptions. To me, for my use, Docker is really a lightweight role-based VM I can put on a provisioned host of some kind [another VM, dedicated server]. In other words, its a really simple way to deploy X identical instances of an entire service/application. If any component in that instance of the application is non functional the container is dead and you route requests to a different container. The problem with the "one process per Docker instance" logic is you need a great deal more service discovery logic as a result. There are a bunch of projects/methods to simplify this but at the end of the day it adds complexity to operations. Instead of a single health check to monitor, you have X containers to monitor. You have X containers to discover, etc. Sure, this lets you get out of running Supervisor and SSH on a Docker container...but I think it adds alot of application complexity that you can avoid 90% of the time. SOA makes sense when you have a large deployment but if you are deploying to a cluster of 5 machines...its overkill. Many [likely the majority] of projects are at the scale of a basic cluster for redundancy.
- vidarh 12y ago
  I think that's just an issue of tooling and getting used to a different way of working with the services. If you run each of those services as a single docker container and group them on the same server, there does not need to be any more service discovery: You can link all of them. That still gives you the flexibility of modifying the linking without having to modify any of the containers. The main practical difference vs. running them all in one vm is explicitly documenting dependencies - both which service relies on which package sets etc. (assuming you're strict about what you put in the Dockerfiles) and which part of your system needs access to which other component or needs to share access to which directories. You can also easily start them e.g. via systemd unit files or via a tool like fleet, and pull the ip/port of dependencies on startup of the container and pass them as arguments. This also makes deploying them as a unit easy, but still decouples it: If you need to move one component to a different server, or suddenly realize your foobar server needs lots of resources and want to load-balance it across multiple machines, you can easily do so by forwarding a port via stunnel or load-balancing via haproxy or what have you without affecting the actual containers. As for your health checks, you can still have a single all-encompassing health check if you want - whether it's in one "vm" or a set of separate containers does not really change that in any way, though the bigger the setup, the more additional checks you'll probably want to add. You have X containers to monitor, but you had X server processes to monitor previously. If your use case made it ok to depend on just e.g. hitting a page on a website that depends on all X services being up, then your single old monitor still does the job. The overhead can be made really minimal and beneficial even for a single server setup, yet help you if/when you suddenly want to scale part of it. Personally, I run about a dozen docker containers on my home server so far, and I expect that to increase several times over as e.g. every web app I experiment with ends up running in its own container, with the docker setup in a git repo as documentation of exactly which project caused me to pull in which additional packages etc., and keeping the host itself as pristine as possible. Makes me lot less nervous about distribution upgrades etc.
yebyen 12y ago
Typo: where a special key with force a specific command
rdtsc 12y ago
What is the general pattern of usage with Docker containers? Are they supposed to isolate just one application and do IO via a networks socket? It used to be you ran everything on a real server. Then it moved to VMs in the cloud. So spawning and managing VM was the the new thing to with a large IaaS,PaaS,...aaS industry around it. The product would consist of one or multiple VMs. All possibly running multiple applications as different processes. Then I guess inter-dependencies between applications and OS versions got to be complicated, so the idea was that each application should run in its own lightweight VM (with LXC as long as they share the same base OS kernel at least). Isn't this just pushing the problems into managing dependencies between more little VMs while also constraining the architecture? It increases the difficulty of synchronizing 10 or 20 separate isolated applications start,upgrade,fail. Maybe all that living on yet another VM guest machine (so conceptually having 2 virtualization levels). Handling more complicated network setup (bridging, firewall rules at multiple levels). Handling effects of disk and other subsystem interacting with each other in strange, sometimes sub-optimal ways. One idea was that ok, this is good for security. One can build secure containers. But doesn't SELinux do that better? It even has a multi-level security mode. It sure is complicated but it is used.
- opendais 12y ago
  > What is the general pattern of usage with Docker containers? Are they supposed to isolate just one application and do IO via a networks socket? The maintainers of Docker appear to strongly believe that 1 application per container is the way to go. At sufficient scale, they are pretty much correct. You grab Y bare metal hosts and spin up X docker containers for X processes. > Isn't this just pushing the problems into managing dependencies between more little VMs while also constraining the architecture? It increases the difficulty of synchronizing 10 or 20 separate isolated applications start,upgrade,fail. Maybe all that living on yet another VM guest machine (so conceptually having 2 virtualization levels). Handling more complicated network setup (bridging, firewall rules at multiple levels). Handling effects of disk and other subsystem interacting with each other in strange, sometimes sub-optimal ways. A handful of people advocate for immutable role-based containers [e.g. https://devopsu.com/blog/docker-misconceptions/ https://devopsu.com/blog/docker-misconceptions/] for that reason. In that use case, its really replacing running something like Xen + Chef, KVM + Ansible, or whatever. You grab a host machine, you stand up your X containers with Z processes per container on Y hosts. I think this is really equivalent to the Microservices vs. Normal SOA vs. Monolithic argument. Given sufficient scale/requirements, each makes sense. However, none of them are optimal for all situations.
lifty 12y ago
Its funny that he uses Docker as a build and installation tool for nsenter. If you look at the installation example from the nsenter github page it shows how you can mount your host's /usr/local/bin inside the container where nsenter will be built. Pretty nifty/hacky way for building software and keeping your system clean.
jaybuff 12y ago
nsenter might be a short term solution, but IMHO, getting a resolution to this issue https://github.com/dotcloud/docker/issues/1228 https://github.com/dotcloud/docker/issues/1228 so you can just do "docker exec <cid> /bin/bash" is much simpler than the nsenter call he recommends. Plus, he leaves out chroot, so I have a different view of the filesystem than PID 1 inside the container does.
drydot 12y ago
i don't see that wrong to setup the ssh service, if i accept ssh daemon is safe which i do.
- diminoten 12y ago
  I recommend you read the submission, then!
- vidarh 12y ago
  You need not just accept that the ssh daemon itself is safe, but that: - Your key management is safe. - The process manager you now need to introduce to start sshd and the app running is safe. - That the ssh daemon is sufficiently protected against abuse. - That your configuration of it is safe. If you don't need ssh in every container to do achieve what you need to achieve, why do you want to have to deal with each of those and waste the extra resources of having a bunch of extra sshd's and process monitors running? (To the last point: Yesterday we suffered an attempt at brute-forcing ssh on a public facing server. We're used to people trying to brute force passwords. But as it happens, it is "easy" to make openssh consume all of your servers resources if you don't block access on the network level in the event of an apparent attack; so if any of those ssh servers are reachable in any way from the outside, you have just increased your attack surface even if your key management and everything else is perfect and they have no way of actually getting in)
  - FooBarWidget 12y ago
    If you are worried about the attack surface, then SSH - as Baseimage-docker configures it - isn't that much of an issue. By default, we do not expose the SSH port to the public Internet, nor do we install any keys. Unless otherwise configured by the user, you first have to login to the host machine, and then from there login to the container through SSH.
    - vidarh 12y ago
      While it's great that you ship with secure defaults, to me, if you're going to restrict it to access from the host only, that just makes it more pointless to run sshd in the containers vs. the alternatives presented in the article.
FooBarWidget 12y ago
I am the author of baseimage-docker (http://phusion.github.io/baseimage-docker/ http://phusion.github.io/baseimage-docker/) and I work at Phusion. I have the feeling that Jerome wrote this article mainly in response to the fact that baseimage-docker encourages using SSH as a way to login to the container. I believe that the ability to login to the container is very important. Depending on how you architect your container, you might not have to, but I believe that it's always good to have the ability to, even if only as a last resort method. I had a pleasant conversation with Jerome quite a while ago about SSH and what the "right" way is to login to a Docker container. We were not able to find consensus, but Jerome is a brilliant guy and his reasons were sound. For some time, I considered using lxc-attach to replace the role of SSH. Unfortunately, a few weeks later, Docker 0.9 came out and no longer used LXC as the default backend, and so suddenly lxc-attach stopped working. We decided to stick with SSH until there's a better way. Solomon Shykes told us that they have plans to introduce an lxc-attach-like tool in Docker core. Unfortunately, as of Docker 1.0.1, this feature still hasn't arrived. Now, Jerome is advocating nsenter. There is currently an ongoing discussion on the baseimage-docker bug tracker about replacing SSH with nsenter: https://github.com/phusion/baseimage-docker/issues/102 https://github.com/phusion/baseimage-docker/issues/102 But leaving all of that aside, we regularly get told by people that Baseimage-docker "misses the point" of Docker. But what is the point of Docker? Some people, including Jerome, believe it's all about microservices and running one process in a container. We take a more balanced, nuanced view. We believe that Docker should be regarded as a flexible tool, that can be mended into whatever you want. You can make single-process microservices, if you want to and if you believe that's the right choice for you. Or you can choose to make multi-process microservices, if that makes sense. Or you can choose to treat Docker like a lightweight VM. We believe that all of those choices are correct. We don't believe that one should ONLY use Docker to build microservices, especially because Microservices Are Not A Free Lunch (http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html http://highscalability.com/blog/2014/4/8/microservices-not-a...). Baseimage-docker is about enabling users to do whatever they want to. It's about choice. It's not about cargo-culting everything into a single philosophy. This is why Baseimage-docker is extremely small and minimalist (only 6 MB memory over), flexible and thoroughly documented. Baseimage-docker is not about advocating treating Docker as heavyweight VMs.
- 12y ago
zobzu 12y ago
its all about useability basically use nsenter. other points in the blog are true but its not what people want. people want nsenter, but they dont know, so they use ssh.
ksikka 12y ago
Volumes are convenient! But what do you do if you have a multi-host setup? Multiple volumes, shared volumes, distributed file systems, NFS? What do you use when?
- mixmastamyk 12y ago
  Would you not mount nfs volume in each container to share data?
  - ksikka 12y ago
    ok, then how do you increase server storage with minimum downtime? say you're in a cloud environment, specifically AWS
  - ksikka 12y ago
    also NFS does not guarantee consistency, which is bad when you're storing server configs in the volume.
x1798DE 12y ago
Reading some of the other comments in this thread, I think I'm getting a better idea of why this guy is suggesting that SSH in Docker could complicate things, especially when you are scaling to a large number of servers, but what I still don't quite understand his argument about security updates. Is that in any way SSH specific, beyond the fact that SSH is powerful and it's critical to be up-to-date on powerful things? If it's a problem to update SSH, then it should also be a problem to update whatever else you have in your Docker container. I guess there's some argument to be made that if you're running a single-purpose Docker container, the updates to whatever service it's running won't sync up with the updates to SSH, so you may drastically increase the number of times you'll have to package the image, but that's just a general argument in favor of single-purpose containers, not anything specific to SSH like the key management issue.
vbit 12y ago
Wow. Given all the hype around docker, I'm really surprised nsenter type functionality isn't part of the core. Glad I'm using FreeBSD where running a shell in your jail is just 'jexec my_jail bash'. Compare that to the rigamarole described in the article. Maybe the docker guys will benefit from just reviewing the tooling for existing systems. I also recommend a look at ezjail-admin.
- vidarh 12y ago
  nsenter is part of util-linux, which means it will eventually make it into pretty much every Linux distro around. The functionality doesn't belong in Docker, because it isn't Docker specific - all it depends on is cgroups, and cgroups is a kernel feature, not a Docker feature. Once it's in there, a "jexec" equivalent would a couple of lines of shell script.
  - vbit 12y ago
    Being part of util-linux makes sense. However, it would be so much nicer if I didn't have to look up those two lines.
atoponce 12y ago
As a cloud hosting provider, I don't want to give customers access to the hardware node. Then what?

125 points by davidkellis 12y ago

Hrrm. I want to agree, but as far as containers go outside of purely docker, I just don't buy these arguments. It makes perfect sense to use industry standard, encrypted communications with proven cryptography when creating vast numbers of systems. Yes, you don't have to. No, that doesn't mean it's a bad idea. I'm not sure why things would be any different with docker, and after reading the article I'm unconvinced. If you want to be locked in to docker's APIs, so be it. If you want to be free of them and integrate in other ways, such as proven, portable, secure methods like SSH, that's fine too. If a management task as simple as deploying a key is so hard in docker (couldn't you just bind-mount a read only .ssh dir?), maybe you should consider alternative methods of container instantiation. I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite.

ldlework 12y ago
I don't think I understand your point. From the Docker Host itself, if you need to manage the state of a container, the intuition is that you need to go into the container (with SSH) in order to do so. But by externalizing your state, you can manage it without the need to enter the container. Assuming your Docker Host is secure, this doesn't make anything less secure just because you're no longer abusing SSHd in order to manage your application's state. In the case you need to gdb, or strace the process, you can do that from the Docker Host with nsenter. Assuming your Docker Host is secure, you no longer need to abuse SSHd to carry out a debugging task that has nothing to do with needing a secure shell. Neither of these use-cases have anything to do with the security of SSH. In the case that you need to do these things from a remote host, the prescribed answer is indeed SSHd to access the Docker Host, at which point you switch to the previously suggested methods for managing state. "I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite." Because now you only have to worry about one security layer instead of N security layers for each container you run. The security layer is now actually coupled to the act of granting access to the host, its intended purpose vs granting access to a container so you can manage its state or debug it or whatever. As far as being locked into Docker's APIs, I totally miss the aim of this remark. Volumes are just paths on the filesystem. If you're talking about the interoperability of standard tools to manage your state, I don't think they will have problems in this case.
- contingencies 12y ago
  the prescribed answer is indeed SSHd to access the Docker Host, at which point you switch to the previously suggested methods for managing state. [...] As far as being locked into Docker's APIs, I totally miss the aim of this remark. Yes, you missed the point. Please read the other response to comprehend the difference.
vidarh 12y ago
> I don't see how granting access to the host is a cleaner architecture... from a security standpoint, it seems the opposite. If you are concerned about granting access to the host, you should be concerned about granting access to a Docker container, especially as root, until it's far more battle tested. And if you are concerned about this, there's a simple solution: Group your Docker containers in KVM vm's. If you are not concerned about this, you can easily do as he suggested, and force "nsenter" on login. Or you can do both: Group your containers in a KVM vm and force ssh into the KVM to use nsenter into the appropriate containers. Part of the point is that it makes little sense to add to the complexity and attack surface to add a process monitor per container (to run both sshd and the app) and an sshd per container, even if what you add is something that provides "industry standard, encrypted communications with proven cryptography" if it is not actually needed inside the containers. You're also not getting "locked into" Dockers API much if at all. What is proposing depends on two things: A way of executing a command that can access a certain part of the containers filesystem, and a way of entering the appropriate namespaces. Both are easily abstracted behind tiny (1-2 line) scripts where you can replace "docker run" and "nsenter" with ssh if your needs change. And one (nsenter) is entirely independent of Docker - it will work with e.g. lxc and anything else that makes use of cgroups as well.
- contingencies 12y ago
  If you are concerned about granting access to the host... Everyone should be. There's a simple solution: Group your Docker containers in KVM vm's. I consider that an ugly hack that should be avoided at all costs. Why? It's not viable without up-front automation investment, ongoing maintenance overheads, and additional latency and inefficiency, for starters. It's also probably less portable, particularly to embedded systems. it makes little sense to add to the complexity and attack surface to add a process monitor per container and an sshd per container Process monitor? What are you talking about? At least with LXC, each container has PID 1 (master application process, init system, whatever) and can be stopped/started easily with lxc-stop -n nameofcontainer. As for sshd, my point was that it can make sense because it's secure, remote-accessible, proven. The article premise was that it's a bad idea, but none of its arguments hold, to my mind. Both are easily abstracted behind tiny (1-2 line) scripts... That aren't remotely accessible, without breaking the abstraction and creating some shared access scenario on the host. The entire point of an abstraction is to be clean. It's not clean if you have to adopt hacky workarounds and dump all of your existing tools in order to play with it.

CraigJPerry 12y ago

I don't get it. What's wrong with putting my SSH public key in the image? Tbh I'd pay more attention to host key regeneration! Easily overlooked. Until there's better investigation tooling available, sshd seems a fairly sensible approach. MAC / SELinux prevents many other poke-inside techniques on enterprise platforms

waffle_ss 12y ago
I don't think it's SSHd in particular that he has a problem with. Docker maintainers do not seem to like the idea of people treating Docker containers as lightweight VMs. They seem to want images to be restricted to running as few processes as possible - ideally one. I think they view it as too hard to scale once you start going down the path of running multiple services in an image - I've seen the "pets vs cattle"[1] analogy used for explaining why. There is an image by Phusion called baseimage-docker[2] which adds SSHd, init, syslog, and cron in an attempt to make Docker containers more like lightweight VMs. But in the #docker channel, I've seen people have issues with it. For example, one person had some /etc/init.d scripts that wouldn't start up (other ones started up fine). Turns out that one of the signals that init was waiting on to start that script was never getting sent (I think it was networking coming online?), and that was just a side effect of how Docker works that couldn't easily be fixed. The Docker maintainers in the channel discouraged using this image for these reasons. [1]: https://groups.google.com/forum/#!msg/docker-user/pNaBYJkmnAA/TsA3b8u5kf0J https://groups.google.com/forum/#!msg/docker-user/pNaBYJkmnA... [2]: http://phusion.github.io/baseimage-docker/ http://phusion.github.io/baseimage-docker/
- opendais 12y ago
  Yes but I bet the majority of people using Docker don't need/want to scale the majority of their projects to 100+ servers which is what the Docker maintainers focus on. It is a good/valid focus for them. However, it is not for everyone.
  - ldlework 12y ago
    I don't scale out to a hundred servers and I still use Docker with separate processes because scaling is not the only advantage to having stateless single-purpose containers.

opendais 12y ago

I think the core flaw with the "no SSHD in Docker, ever" reasoning is it makes a number of assumptions. To me, for my use, Docker is really a lightweight role-based VM I can put on a provisioned host of some kind [another VM, dedicated server]. In other words, its a really simple way to deploy X identical instances of an entire service/application. If any component in that instance of the application is non functional the container is dead and you route requests to a different container. The problem with the "one process per Docker instance" logic is you need a great deal more service discovery logic as a result. There are a bunch of projects/methods to simplify this but at the end of the day it adds complexity to operations. Instead of a single health check to monitor, you have X containers to monitor. You have X containers to discover, etc. Sure, this lets you get out of running Supervisor and SSH on a Docker container...but I think it adds alot of application complexity that you can avoid 90% of the time. SOA makes sense when you have a large deployment but if you are deploying to a cluster of 5 machines...its overkill. Many [likely the majority] of projects are at the scale of a basic cluster for redundancy.

vidarh 12y ago
I think that's just an issue of tooling and getting used to a different way of working with the services. If you run each of those services as a single docker container and group them on the same server, there does not need to be any more service discovery: You can link all of them. That still gives you the flexibility of modifying the linking without having to modify any of the containers. The main practical difference vs. running them all in one vm is explicitly documenting dependencies - both which service relies on which package sets etc. (assuming you're strict about what you put in the Dockerfiles) and which part of your system needs access to which other component or needs to share access to which directories. You can also easily start them e.g. via systemd unit files or via a tool like fleet, and pull the ip/port of dependencies on startup of the container and pass them as arguments. This also makes deploying them as a unit easy, but still decouples it: If you need to move one component to a different server, or suddenly realize your foobar server needs lots of resources and want to load-balance it across multiple machines, you can easily do so by forwarding a port via stunnel or load-balancing via haproxy or what have you without affecting the actual containers. As for your health checks, you can still have a single all-encompassing health check if you want - whether it's in one "vm" or a set of separate containers does not really change that in any way, though the bigger the setup, the more additional checks you'll probably want to add. You have X containers to monitor, but you had X server processes to monitor previously. If your use case made it ok to depend on just e.g. hitting a page on a website that depends on all X services being up, then your single old monitor still does the job. The overhead can be made really minimal and beneficial even for a single server setup, yet help you if/when you suddenly want to scale part of it. Personally, I run about a dozen docker containers on my home server so far, and I expect that to increase several times over as e.g. every web app I experiment with ends up running in its own container, with the docker setup in a git repo as documentation of exactly which project caused me to pull in which additional packages etc., and keeping the host itself as pristine as possible. Makes me lot less nervous about distribution upgrades etc.

yebyen 12y ago

Typo: where a special key with force a specific command

rdtsc 12y ago

What is the general pattern of usage with Docker containers? Are they supposed to isolate just one application and do IO via a networks socket? It used to be you ran everything on a real server. Then it moved to VMs in the cloud. So spawning and managing VM was the the new thing to with a large IaaS,PaaS,...aaS industry around it. The product would consist of one or multiple VMs. All possibly running multiple applications as different processes. Then I guess inter-dependencies between applications and OS versions got to be complicated, so the idea was that each application should run in its own lightweight VM (with LXC as long as they share the same base OS kernel at least). Isn't this just pushing the problems into managing dependencies between more little VMs while also constraining the architecture? It increases the difficulty of synchronizing 10 or 20 separate isolated applications start,upgrade,fail. Maybe all that living on yet another VM guest machine (so conceptually having 2 virtualization levels). Handling more complicated network setup (bridging, firewall rules at multiple levels). Handling effects of disk and other subsystem interacting with each other in strange, sometimes sub-optimal ways. One idea was that ok, this is good for security. One can build secure containers. But doesn't SELinux do that better? It even has a multi-level security mode. It sure is complicated but it is used.

opendais 12y ago
> What is the general pattern of usage with Docker containers? Are they supposed to isolate just one application and do IO via a networks socket? The maintainers of Docker appear to strongly believe that 1 application per container is the way to go. At sufficient scale, they are pretty much correct. You grab Y bare metal hosts and spin up X docker containers for X processes. > Isn't this just pushing the problems into managing dependencies between more little VMs while also constraining the architecture? It increases the difficulty of synchronizing 10 or 20 separate isolated applications start,upgrade,fail. Maybe all that living on yet another VM guest machine (so conceptually having 2 virtualization levels). Handling more complicated network setup (bridging, firewall rules at multiple levels). Handling effects of disk and other subsystem interacting with each other in strange, sometimes sub-optimal ways. A handful of people advocate for immutable role-based containers [e.g. https://devopsu.com/blog/docker-misconceptions/ https://devopsu.com/blog/docker-misconceptions/] for that reason. In that use case, its really replacing running something like Xen + Chef, KVM + Ansible, or whatever. You grab a host machine, you stand up your X containers with Z processes per container on Y hosts. I think this is really equivalent to the Microservices vs. Normal SOA vs. Monolithic argument. Given sufficient scale/requirements, each makes sense. However, none of them are optimal for all situations.

lifty 12y ago

Its funny that he uses Docker as a build and installation tool for nsenter. If you look at the installation example from the nsenter github page it shows how you can mount your host's /usr/local/bin inside the container where nsenter will be built. Pretty nifty/hacky way for building software and keeping your system clean.

jaybuff 12y ago

nsenter might be a short term solution, but IMHO, getting a resolution to this issue https://github.com/dotcloud/docker/issues/1228 https://github.com/dotcloud/docker/issues/1228 so you can just do "docker exec <cid> /bin/bash" is much simpler than the nsenter call he recommends. Plus, he leaves out chroot, so I have a different view of the filesystem than PID 1 inside the container does.

drydot 12y ago

i don't see that wrong to setup the ssh service, if i accept ssh daemon is safe which i do.

diminoten 12y ago
I recommend you read the submission, then!
vidarh 12y ago
You need not just accept that the ssh daemon itself is safe, but that: - Your key management is safe. - The process manager you now need to introduce to start sshd and the app running is safe. - That the ssh daemon is sufficiently protected against abuse. - That your configuration of it is safe. If you don't need ssh in every container to do achieve what you need to achieve, why do you want to have to deal with each of those and waste the extra resources of having a bunch of extra sshd's and process monitors running? (To the last point: Yesterday we suffered an attempt at brute-forcing ssh on a public facing server. We're used to people trying to brute force passwords. But as it happens, it is "easy" to make openssh consume all of your servers resources if you don't block access on the network level in the event of an apparent attack; so if any of those ssh servers are reachable in any way from the outside, you have just increased your attack surface even if your key management and everything else is perfect and they have no way of actually getting in)
- FooBarWidget 12y ago
  If you are worried about the attack surface, then SSH - as Baseimage-docker configures it - isn't that much of an issue. By default, we do not expose the SSH port to the public Internet, nor do we install any keys. Unless otherwise configured by the user, you first have to login to the host machine, and then from there login to the container through SSH.
  - vidarh 12y ago
    While it's great that you ship with secure defaults, to me, if you're going to restrict it to access from the host only, that just makes it more pointless to run sshd in the containers vs. the alternatives presented in the article.

FooBarWidget 12y ago

I am the author of baseimage-docker (http://phusion.github.io/baseimage-docker/ http://phusion.github.io/baseimage-docker/) and I work at Phusion. I have the feeling that Jerome wrote this article mainly in response to the fact that baseimage-docker encourages using SSH as a way to login to the container. I believe that the ability to login to the container is very important. Depending on how you architect your container, you might not have to, but I believe that it's always good to have the ability to, even if only as a last resort method. I had a pleasant conversation with Jerome quite a while ago about SSH and what the "right" way is to login to a Docker container. We were not able to find consensus, but Jerome is a brilliant guy and his reasons were sound. For some time, I considered using lxc-attach to replace the role of SSH. Unfortunately, a few weeks later, Docker 0.9 came out and no longer used LXC as the default backend, and so suddenly lxc-attach stopped working. We decided to stick with SSH until there's a better way. Solomon Shykes told us that they have plans to introduce an lxc-attach-like tool in Docker core. Unfortunately, as of Docker 1.0.1, this feature still hasn't arrived. Now, Jerome is advocating nsenter. There is currently an ongoing discussion on the baseimage-docker bug tracker about replacing SSH with nsenter: https://github.com/phusion/baseimage-docker/issues/102 https://github.com/phusion/baseimage-docker/issues/102 But leaving all of that aside, we regularly get told by people that Baseimage-docker "misses the point" of Docker. But what is the point of Docker? Some people, including Jerome, believe it's all about microservices and running one process in a container. We take a more balanced, nuanced view. We believe that Docker should be regarded as a flexible tool, that can be mended into whatever you want. You can make single-process microservices, if you want to and if you believe that's the right choice for you. Or you can choose to make multi-process microservices, if that makes sense. Or you can choose to treat Docker like a lightweight VM. We believe that all of those choices are correct. We don't believe that one should ONLY use Docker to build microservices, especially because Microservices Are Not A Free Lunch (http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html http://highscalability.com/blog/2014/4/8/microservices-not-a...). Baseimage-docker is about enabling users to do whatever they want to. It's about choice. It's not about cargo-culting everything into a single philosophy. This is why Baseimage-docker is extremely small and minimalist (only 6 MB memory over), flexible and thoroughly documented. Baseimage-docker is not about advocating treating Docker as heavyweight VMs.

12y ago

zobzu 12y ago

its all about useability basically use nsenter. other points in the blog are true but its not what people want. people want nsenter, but they dont know, so they use ssh.

ksikka 12y ago

Volumes are convenient! But what do you do if you have a multi-host setup? Multiple volumes, shared volumes, distributed file systems, NFS? What do you use when?

mixmastamyk 12y ago
Would you not mount nfs volume in each container to share data?
- ksikka 12y ago
  ok, then how do you increase server storage with minimum downtime? say you're in a cloud environment, specifically AWS
- ksikka 12y ago
  also NFS does not guarantee consistency, which is bad when you're storing server configs in the volume.

x1798DE 12y ago

Reading some of the other comments in this thread, I think I'm getting a better idea of why this guy is suggesting that SSH in Docker could complicate things, especially when you are scaling to a large number of servers, but what I still don't quite understand his argument about security updates. Is that in any way SSH specific, beyond the fact that SSH is powerful and it's critical to be up-to-date on powerful things? If it's a problem to update SSH, then it should also be a problem to update whatever else you have in your Docker container. I guess there's some argument to be made that if you're running a single-purpose Docker container, the updates to whatever service it's running won't sync up with the updates to SSH, so you may drastically increase the number of times you'll have to package the image, but that's just a general argument in favor of single-purpose containers, not anything specific to SSH like the key management issue.

vbit 12y ago

Wow. Given all the hype around docker, I'm really surprised nsenter type functionality isn't part of the core. Glad I'm using FreeBSD where running a shell in your jail is just 'jexec my_jail bash'. Compare that to the rigamarole described in the article. Maybe the docker guys will benefit from just reviewing the tooling for existing systems. I also recommend a look at ezjail-admin.

vidarh 12y ago
nsenter is part of util-linux, which means it will eventually make it into pretty much every Linux distro around. The functionality doesn't belong in Docker, because it isn't Docker specific - all it depends on is cgroups, and cgroups is a kernel feature, not a Docker feature. Once it's in there, a "jexec" equivalent would a couple of lines of shell script.
- vbit 12y ago
  Being part of util-linux makes sense. However, it would be so much nicer if I didn't have to look up those two lines.

atoponce 12y ago

As a cloud hosting provider, I don't want to give customers access to the hardware node. Then what?