8 ms·

So basically it is just the usage of particular API the underlying OS provides.

by zerr 9y ago

DannyB2 9y ago
A container is launching a process in a way that the kernel lets it believe it can be root, have any illusion of a filesystem and network connectivity that you want it to believe, and be fooled into thinking it is the only process running on this kernel. Therefore, it is as light weight as launching a process? Is that about right?
- eftychis 9y ago
  Pretty much. Usually you run your own init that starts the desired process. If you as a process have id (pid) 1 then you need to manage orphaned processes (and zombies) and general signal handling (e.g. SIGTERM). There is a lot of cgroups and capabilities tweaking involved also. You can also use bpf (e.g. seccomp-bpf to restrict system calls etc. In general you i) add the illusion of king in your little own world ii) provide necessary handling that the kernel expects from a king iii) restrict resources (e.g. memory, system calls,...) and facilitate communication with the rest of the worlds.
  - zenlikethat 9y ago
    There is also now a `--init` flag for docker run and for dockerd that will run a Pretty Reasonable PID1 (tini) automatically.
ntnn 9y ago
Yes, I've given talks on what containers actually are and which technologies they combine. Containers oldest technology would be mount namespaces from 2002 - the youngest are user namespaces from 2013 (on linux anyway, solaris and IBM had containers before). People are always amazed that container products are actually mostly just the glue around what the kernel already provides and that their history goes back almost two decades.
- tyingq 9y ago
  Maybe even further. Mainframes have had similar for decades. Yes, perhaps more like actual VM's, but the namespacing, cgroup stuff, network overlays, etc, are very similar. My mainframe folks, when talking about docker, universally yawn.
  - icebraining 9y ago
    So do the Linux guys. Virtuozzo (later OpenVZ) has been around for almost twenty years.
- Huggernaut 9y ago
  cgroup namespaces are a little younger and came arrived in 4.6 I think http://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html http://man7.org/linux/man-pages/man7/cgroup_namespaces.7.htm...
  - ntnn 9y ago
    cgroups v2 probably, cgroups v1 is definitely older.
    - Huggernaut 9y ago
      Nah, I'm talking about the cgroup _namespace_ specifically, not cgroups in general.
erikb 9y ago
That's why some people believe that in 10 years we won't have docker, we won't have kubernetes, but that it will be intuitively integrated into the OS through new system design patterns that still have to appear out of all the crazy experiments we're doing.
- hodgesrm 9y ago
  That might be true of Docker. Kubernetes on the other hand is a way to deploy and manage distributed applications, which are by definition bigger than any single OS.
  - monocasa 9y ago
    Not if the OS starts to intrinsically view itself as a node in a distributed system. Mainframes sort of see SMP and separate nodes connected over a network as two sides of the same coin, just loosely vs. tightly coupled. https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0/com.ibm.admin.doc/ids_admin_1049.htm https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0...
    - icebraining 9y ago
      Also Plan9: "Since CPU servers and terminals use the same kernel, users may choose to run programs locally on their terminals or remotely on CPU servers. The organization of Plan 9 hides the details of system connectivity allowing both users and administrators to configure their environment to be as distributed or centralized as they wish. Simple commands support the construction of a locally represented name space spanning many machines and networks. At work, users tend to use their terminals like workstations, running interactive programs locally and reserving the CPU servers for data or compute intensive jobs such as compiling and computing chess endgames. At home or when connected over a slow network, users tend to do most work on the CPU server to minimize traffic on the slow links. The goal of the network organization is to provide the same environment to the user wherever resources are used." http://doc.cat-v.org/plan_9/4th_edition/papers/net/ http://doc.cat-v.org/plan_9/4th_edition/papers/net/
  - 9y ago
devmunchies 9y ago
Can someone explain why starting a docker container usually has a 200-300ms start up penalty? Its fine if I want to start a web server but is long if its a just a precompiled script that runs on-command and then dies. Is it Docker that is taking 200ms to start the container or just the nature of the OS APIs?
- tyingq 9y ago
  Could be as simple as the init() process. Have you tried a slimmed down guest distro like Alpine and compared startup times?
- zenlikethat 9y ago
  Mostly it's Docker-specific, but all of the things causing this latency tend to be what makes Docker useful in the first place. In particular, networking and mounts. For starters, if you're just running the typical Docker set up to talk back and forth using a UNIX or TCP socket, all of _that_ networking and HTTP/JSON encoding+decoding will add some overhead. This is even worse if you're going through a proxy layer like the socket which Docker for Mac creates and forwards to the Linux VM. OK, so there's at least a few milliseconds of latency built in for this, the socket forwarding hop on my Docker for Mac seems to add ~5-15ms for instance. Once the daemon receives the request, runc, the command that actually does the work to create a container without all that hoopla, is invoked. Then, to actually start up a container quite a few mounts get set up either for devices (potentially including tty in the case of `run -t`) or for the union filesystem. Take a look at the output of `mount` in a container sometime, around 30-40 mounts (and all those overlay layers...) and a lot of that gets set up on the fly because each container has its own unique mount namespace and view of the world. Then after all of that, you need to set up networking too (`docker run --net none` will skip this), otherwise containers can't talk to the Internet, to each other, or listen on ports of the host. Remember, each has its own network namespace, created from scratch. So, Docker's doing all manner of adding port proxies, modifying iptables, attaching containers to interfaces, and so on. Otherwise, `docker run alpine wget -qO- https://google.com` https://google.com` or `docker run --net=internalnet` wouldn't work out of the box. The work isn't easily done concurrently since there are so many dependencies - e.g., you need to have a process in order to set a network namespace, in order to run that process, you need to have a root FS prepared to pivot root into, and so on. Meanwhile to all of this happening the Docker daemon is doing god knows what, it could be pulling images, or running other containers already, supervising processes, etc. While many of those things are concurrent due to use of goroutines, they do eat up resources and generally too much work saturates and slows down the daemon. That's not really unique to Docker though, any program doing all this stuff at once would have that issue. Anyway, that's why Docker takes way longer to start a process than good old execve() in your shell.
tyingq 9y ago
Yup. It's all namespaces and cgroups. The base "docker" command adds a little value with os respositories, network overlays, etc, but not a lot. Personally, I think K8S with their cri-o (https://github.com/kubernetes-incubator/cri-o/blob/master/README.md https://github.com/kubernetes-incubator/cri-o/blob/master/RE...) runtime is going to eventually eat Docker's lunch. As a company, Docker needs Swarm/Enterprise to take off in order to have a differentiator. That isn't happening. Kubernetes, on the other hand, just needs cri-o (mostly done) and an image repository / builder to kill off Docker. See this for how small the gap really is: https://github.com/p8952/bocker/blob/master/README.md https://github.com/p8952/bocker/blob/master/README.md Basically, Docker has almost no moat. They have mostly only brand recognition. Not discounting their efforts, but they are playing checkers and Google is playing chess.
- hardwaresofton 9y ago
  I think this is how I most commonly recognize someone that is familiar with lxc +/- docker and containerization in general. A firm grasp of the fact that they're just beefy processes, isolated wiht namespaces and cgroups is the best, most succint way to describe docker (without even mentioning the benefits), but it also requires that the hearer knows what namespaces and cgroups are. If I had to rank understanding in explanations of docker: 1. "Makes your application really portable"/other explanations that only cover the benefits of docker not how it works 2. "Lightweight VMs" 3. "LXC + some other stuff" 4. "processes + namespaces + cgroups (+/- image management tools, etc)" 100% agreed on the point you made, pretty sure docker+swarm and other orchestration efforts have basically lost the competition. Kubernetes just has the mind share, and even better than that -- it's actually good. Kubernetes also has multiple competing container runtimes all rushing to fit the CRI (Container Runtime Interface). Just some of the stuff out there: * cri-containerd - https://github.com/containerd/cri-containerd https://github.com/containerd/cri-containerd * runv (hypervisor based) - https://github.com/hyperhq/runv https://github.com/hyperhq/runv * clear containers (hypervisor based) - https://github.com/clearcontainers/runtime https://github.com/clearcontainers/runtime * kata containers (hypervisor based, collaboration of runv + clear container) - https://katacontainers.io/ https://katacontainers.io/ * frakti (combination of runv & docker, enabling switching @ runtime) - https://github.com/kubernetes/frakti https://github.com/kubernetes/frakti A bunch of the projects are in their infancy, but the CNCF/kubernetes and the community are doing the right thing investing in lots of them, and letting the good ones bubble to the top. I don't run a production kubernetes cluster but I've found cri-containerd to be really easy to use and haven't had any major problems with it, mostly small configuration things. The last one, frakti, gets me really excited, because it lets you be flexible about which containers run in more protected environments and which don't. Also really exciting is frakti v2 (https://github.com/kubernetes/frakti/tree/containerd-kata https://github.com/kubernetes/frakti/tree/containerd-kata), which is kata + containerd.

So basically it is just the usage of particular API the underlying OS provides.

by zerr 9y ago

So basically it is just the usage of particular API the underlying OS provides.

A container is launching a process in a way that the kernel lets it believe it can be root, have any illusion of a filesystem and network connectivity that you want it to believe, and be fooled into thinking it is the only process running on this kernel. Therefore, it is as light weight as launching a process? Is that about right?

eftychis 9y ago
Pretty much. Usually you run your own init that starts the desired process. If you as a process have id (pid) 1 then you need to manage orphaned processes (and zombies) and general signal handling (e.g. SIGTERM). There is a lot of cgroups and capabilities tweaking involved also. You can also use bpf (e.g. seccomp-bpf to restrict system calls etc. In general you i) add the illusion of king in your little own world ii) provide necessary handling that the kernel expects from a king iii) restrict resources (e.g. memory, system calls,...) and facilitate communication with the rest of the worlds.
- zenlikethat 9y ago
  There is also now a `--init` flag for docker run and for dockerd that will run a Pretty Reasonable PID1 (tini) automatically.

ntnn 9y ago

Yes, I've given talks on what containers actually are and which technologies they combine. Containers oldest technology would be mount namespaces from 2002 - the youngest are user namespaces from 2013 (on linux anyway, solaris and IBM had containers before). People are always amazed that container products are actually mostly just the glue around what the kernel already provides and that their history goes back almost two decades.

tyingq 9y ago
Maybe even further. Mainframes have had similar for decades. Yes, perhaps more like actual VM's, but the namespacing, cgroup stuff, network overlays, etc, are very similar. My mainframe folks, when talking about docker, universally yawn.
- icebraining 9y ago
  So do the Linux guys. Virtuozzo (later OpenVZ) has been around for almost twenty years.
Huggernaut 9y ago
cgroup namespaces are a little younger and came arrived in 4.6 I think http://man7.org/linux/man-pages/man7/cgroup_namespaces.7.html http://man7.org/linux/man-pages/man7/cgroup_namespaces.7.htm...
- ntnn 9y ago
  cgroups v2 probably, cgroups v1 is definitely older.
  - Huggernaut 9y ago
    Nah, I'm talking about the cgroup _namespace_ specifically, not cgroups in general.

erikb 9y ago

That's why some people believe that in 10 years we won't have docker, we won't have kubernetes, but that it will be intuitively integrated into the OS through new system design patterns that still have to appear out of all the crazy experiments we're doing.

hodgesrm 9y ago
That might be true of Docker. Kubernetes on the other hand is a way to deploy and manage distributed applications, which are by definition bigger than any single OS.
- monocasa 9y ago
  Not if the OS starts to intrinsically view itself as a node in a distributed system. Mainframes sort of see SMP and separate nodes connected over a network as two sides of the same coin, just loosely vs. tightly coupled. https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0/com.ibm.admin.doc/ids_admin_1049.htm https://www.ibm.com/support/knowledgecenter/en/SSGU8G_12.1.0...
  - icebraining 9y ago
    Also Plan9: "Since CPU servers and terminals use the same kernel, users may choose to run programs locally on their terminals or remotely on CPU servers. The organization of Plan 9 hides the details of system connectivity allowing both users and administrators to configure their environment to be as distributed or centralized as they wish. Simple commands support the construction of a locally represented name space spanning many machines and networks. At work, users tend to use their terminals like workstations, running interactive programs locally and reserving the CPU servers for data or compute intensive jobs such as compiling and computing chess endgames. At home or when connected over a slow network, users tend to do most work on the CPU server to minimize traffic on the slow links. The goal of the network organization is to provide the same environment to the user wherever resources are used." http://doc.cat-v.org/plan_9/4th_edition/papers/net/ http://doc.cat-v.org/plan_9/4th_edition/papers/net/
- 9y ago

devmunchies 9y ago

Can someone explain why starting a docker container usually has a 200-300ms start up penalty? Its fine if I want to start a web server but is long if its a just a precompiled script that runs on-command and then dies. Is it Docker that is taking 200ms to start the container or just the nature of the OS APIs?

tyingq 9y ago
Could be as simple as the init() process. Have you tried a slimmed down guest distro like Alpine and compared startup times?
zenlikethat 9y ago
Mostly it's Docker-specific, but all of the things causing this latency tend to be what makes Docker useful in the first place. In particular, networking and mounts. For starters, if you're just running the typical Docker set up to talk back and forth using a UNIX or TCP socket, all of _that_ networking and HTTP/JSON encoding+decoding will add some overhead. This is even worse if you're going through a proxy layer like the socket which Docker for Mac creates and forwards to the Linux VM. OK, so there's at least a few milliseconds of latency built in for this, the socket forwarding hop on my Docker for Mac seems to add ~5-15ms for instance. Once the daemon receives the request, runc, the command that actually does the work to create a container without all that hoopla, is invoked. Then, to actually start up a container quite a few mounts get set up either for devices (potentially including tty in the case of `run -t`) or for the union filesystem. Take a look at the output of `mount` in a container sometime, around 30-40 mounts (and all those overlay layers...) and a lot of that gets set up on the fly because each container has its own unique mount namespace and view of the world. Then after all of that, you need to set up networking too (`docker run --net none` will skip this), otherwise containers can't talk to the Internet, to each other, or listen on ports of the host. Remember, each has its own network namespace, created from scratch. So, Docker's doing all manner of adding port proxies, modifying iptables, attaching containers to interfaces, and so on. Otherwise, `docker run alpine wget -qO- https://google.com` https://google.com` or `docker run --net=internalnet` wouldn't work out of the box. The work isn't easily done concurrently since there are so many dependencies - e.g., you need to have a process in order to set a network namespace, in order to run that process, you need to have a root FS prepared to pivot root into, and so on. Meanwhile to all of this happening the Docker daemon is doing god knows what, it could be pulling images, or running other containers already, supervising processes, etc. While many of those things are concurrent due to use of goroutines, they do eat up resources and generally too much work saturates and slows down the daemon. That's not really unique to Docker though, any program doing all this stuff at once would have that issue. Anyway, that's why Docker takes way longer to start a process than good old execve() in your shell.

tyingq 9y ago

Yup. It's all namespaces and cgroups. The base "docker" command adds a little value with os respositories, network overlays, etc, but not a lot. Personally, I think K8S with their cri-o (https://github.com/kubernetes-incubator/cri-o/blob/master/README.md https://github.com/kubernetes-incubator/cri-o/blob/master/RE...) runtime is going to eventually eat Docker's lunch. As a company, Docker needs Swarm/Enterprise to take off in order to have a differentiator. That isn't happening. Kubernetes, on the other hand, just needs cri-o (mostly done) and an image repository / builder to kill off Docker. See this for how small the gap really is: https://github.com/p8952/bocker/blob/master/README.md https://github.com/p8952/bocker/blob/master/README.md Basically, Docker has almost no moat. They have mostly only brand recognition. Not discounting their efforts, but they are playing checkers and Google is playing chess.

hardwaresofton 9y ago
I think this is how I most commonly recognize someone that is familiar with lxc +/- docker and containerization in general. A firm grasp of the fact that they're just beefy processes, isolated wiht namespaces and cgroups is the best, most succint way to describe docker (without even mentioning the benefits), but it also requires that the hearer knows what namespaces and cgroups are. If I had to rank understanding in explanations of docker: 1. "Makes your application really portable"/other explanations that only cover the benefits of docker not how it works 2. "Lightweight VMs" 3. "LXC + some other stuff" 4. "processes + namespaces + cgroups (+/- image management tools, etc)" 100% agreed on the point you made, pretty sure docker+swarm and other orchestration efforts have basically lost the competition. Kubernetes just has the mind share, and even better than that -- it's actually good. Kubernetes also has multiple competing container runtimes all rushing to fit the CRI (Container Runtime Interface). Just some of the stuff out there: * cri-containerd - https://github.com/containerd/cri-containerd https://github.com/containerd/cri-containerd * runv (hypervisor based) - https://github.com/hyperhq/runv https://github.com/hyperhq/runv * clear containers (hypervisor based) - https://github.com/clearcontainers/runtime https://github.com/clearcontainers/runtime * kata containers (hypervisor based, collaboration of runv + clear container) - https://katacontainers.io/ https://katacontainers.io/ * frakti (combination of runv & docker, enabling switching @ runtime) - https://github.com/kubernetes/frakti https://github.com/kubernetes/frakti A bunch of the projects are in their infancy, but the CNCF/kubernetes and the community are doing the right thing investing in lots of them, and letting the good ones bubble to the top. I don't run a production kubernetes cluster but I've found cri-containerd to be really easy to use and haven't had any major problems with it, mostly small configuration things. The last one, frakti, gets me really excited, because it lets you be flexible about which containers run in more protected environments and which don't. Also really exciting is frakti v2 (https://github.com/kubernetes/frakti/tree/containerd-kata https://github.com/kubernetes/frakti/tree/containerd-kata), which is kata + containerd.