4 ms·

containers as an expanded chroot jail. Could you point me to a succinct description of exactly the semantics of containers as an expanded chroot jail? I've bee

by mafribe 10y ago

containers as an expanded chroot jail. Could you point me to a succinct description of exactly the semantics of containers as an expanded chroot jail? I've been looking, but have so far not found anything.

geofft 10y ago
The term "container" (on Linux) refers to the high-level combination of two concrete low-level interfaces: namespaces and control groups (cgroups). I like to think of namespaces as converting certain global variables in the kernel to local variables for each process. These local variables are then inherited to child processes. chroots are the simplest example, although they predate namespaces. Ordinarily you'd think of a system as having a single root directory; somewhere in the kernel is a global variable DIR root. But in fact, each process has its own root directory pointer in the process structure. Most of those pointers have the same value in every process, but if you run chroot, you change that pointer for the current process and all its children. The list of possible namespaces is in the clone(2) and unshare(2) manpages (`man 2 clone` and `man 2 unshare`); look for the options starting with CLONE_NEW. They all change some pointer in the process structure, either to a substructure of the original pointer (like chroot does), a deep copy of the structure, or to a new, empty structure. CLONE_NEWIPC changes the pointer for routing System V IPCs. CLONE_NEWNET changes the pointer for the list of network device to a new structure with just a new loopback interface. CLONE_NEWNS copies the mount table instead of keeping a pointer to it, so your process can unmount filesystems without affecting the rest of the system, or vice versa. CLONE_NEWPID changes the pointer to pid 1 / the process ID table to point to yourself (effectively a chroot for process IDs). CLONE_NEWUSER changes the interpretation of user IDs, so you can have UID 0 in your process be a non-zero UID in the outside system. CLONE_NEWUTS creates a copy of the structure containing the machine's hostname, instead of keeping a pointer to the global structure. cgroups are resource control. They let you say that a certain process tree has some maximum amount of RAM, or CPU shares, or so forth. This is useful for making containers perform the way you want, but doesn't really affect their semantics.
- cesnja 10y ago
  But do cgroups really allow setting maximum CPU shares? man 7 cgroups says this about the cpu subsystem: > Cgroups can be guaranteed a minimum number of "CPU shares" when a system is busy. This does not limit a cgroup's CPU usage if the CPUs are not busy. Granted my english isn't the best, but this doesn't seem to indicate any throttling.
  - geofft 10y ago
    I'm not very familiar with cgroups, but the documentation mention in that manpage ( https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt https://www.kernel.org/doc/Documentation/scheduler/sched-bwc... ) talks about throttling and maximum CPU usage. I'd guess that it's just a matter of how you view it - putting a minimum number of CPU shares on the root cgroup is the same as putting a maximum on the rest of the cgroups, right? But maybe one or the other documentation is wrong.
    - cesnja 10y ago
      Both manpage and documentation are probably right, though I'd expect CPU throttling to be significant enough at least to be mentioned. I'm not so sure about your guess, because that apparently works only when CPU is fully utilized. I'm not so sure about my reading comprehension either.
- mafribe 10y ago
  Thanks "geofft" and "cesnja". converting certain global variables How do the copies of the variables that become local relate to the originals? What happens at read/write, what happens if the host OS changes them? In other words, what is the semantics of sharing/copying these variables?
  - geofft 10y ago
    It depends on the thing you're unsharing. For a chroot, you get a pointer to a subdirectory of the host root directory. Changes within that directory are visible in both directions. CLONE_NEWPID and CLONE_NEWUSER work similarly; every process has a PID and a UID outside of the container (that is, in the root namespace), but a subset of PIDs and UIDs are visible in the container, with their own values. Creating a process in a PID namespace causes it to get a PID counting from 1 in that namespace, as well as a PID counting from 1 in the parent namespace. A user account in a user namespace has a value (which could be 0) in the namespace, as well as a mapped value in the parent namespace. CLONE_NEWIPC and CLONE_NEWNET create new, empty structures for the IPC namespace and network stack. Changes in one namespace aren't visible to another. You can move network devices between namespaces by using `ip link set dev eth1 netns 1234`, which will move eth1 out of the current process's network namespace and into process 1234's namespace. (This is occasionally useful with physical devices, but more useful with virtual devices like veth and macvlan.) CLONE_NEWNS and CLONE_NEWUTS create a deep copy of the current namespace's mount table and hostname/domainname strings, respectively. Further changes in one namespace do not affect the other.

by mafribe 10y ago

geofft 10y ago

The term "container" (on Linux) refers to the high-level combination of two concrete low-level interfaces: namespaces and control groups (cgroups). I like to think of namespaces as converting certain global variables in the kernel to local variables for each process. These local variables are then inherited to child processes. chroots are the simplest example, although they predate namespaces. Ordinarily you'd think of a system as having a single root directory; somewhere in the kernel is a global variable DIR root. But in fact, each process has its own root directory pointer in the process structure. Most of those pointers have the same value in every process, but if you run chroot, you change that pointer for the current process and all its children. The list of possible namespaces is in the clone(2) and unshare(2) manpages (`man 2 clone` and `man 2 unshare`); look for the options starting with CLONE_NEW. They all change some pointer in the process structure, either to a substructure of the original pointer (like chroot does), a deep copy of the structure, or to a new, empty structure. CLONE_NEWIPC changes the pointer for routing System V IPCs. CLONE_NEWNET changes the pointer for the list of network device to a new structure with just a new loopback interface. CLONE_NEWNS copies the mount table instead of keeping a pointer to it, so your process can unmount filesystems without affecting the rest of the system, or vice versa. CLONE_NEWPID changes the pointer to pid 1 / the process ID table to point to yourself (effectively a chroot for process IDs). CLONE_NEWUSER changes the interpretation of user IDs, so you can have UID 0 in your process be a non-zero UID in the outside system. CLONE_NEWUTS creates a copy of the structure containing the machine's hostname, instead of keeping a pointer to the global structure. cgroups are resource control. They let you say that a certain process tree has some maximum amount of RAM, or CPU shares, or so forth. This is useful for making containers perform the way you want, but doesn't really affect their semantics.

cesnja 10y ago
But do cgroups really allow setting maximum CPU shares? man 7 cgroups says this about the cpu subsystem: > Cgroups can be guaranteed a minimum number of "CPU shares" when a system is busy. This does not limit a cgroup's CPU usage if the CPUs are not busy. Granted my english isn't the best, but this doesn't seem to indicate any throttling.
- geofft 10y ago
  I'm not very familiar with cgroups, but the documentation mention in that manpage ( https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt https://www.kernel.org/doc/Documentation/scheduler/sched-bwc... ) talks about throttling and maximum CPU usage. I'd guess that it's just a matter of how you view it - putting a minimum number of CPU shares on the root cgroup is the same as putting a maximum on the rest of the cgroups, right? But maybe one or the other documentation is wrong.
  - cesnja 10y ago
    Both manpage and documentation are probably right, though I'd expect CPU throttling to be significant enough at least to be mentioned. I'm not so sure about your guess, because that apparently works only when CPU is fully utilized. I'm not so sure about my reading comprehension either.
mafribe 10y ago
Thanks "geofft" and "cesnja". converting certain global variables How do the copies of the variables that become local relate to the originals? What happens at read/write, what happens if the host OS changes them? In other words, what is the semantics of sharing/copying these variables?
- geofft 10y ago
  It depends on the thing you're unsharing. For a chroot, you get a pointer to a subdirectory of the host root directory. Changes within that directory are visible in both directions. CLONE_NEWPID and CLONE_NEWUSER work similarly; every process has a PID and a UID outside of the container (that is, in the root namespace), but a subset of PIDs and UIDs are visible in the container, with their own values. Creating a process in a PID namespace causes it to get a PID counting from 1 in that namespace, as well as a PID counting from 1 in the parent namespace. A user account in a user namespace has a value (which could be 0) in the namespace, as well as a mapped value in the parent namespace. CLONE_NEWIPC and CLONE_NEWNET create new, empty structures for the IPC namespace and network stack. Changes in one namespace aren't visible to another. You can move network devices between namespaces by using `ip link set dev eth1 netns 1234`, which will move eth1 out of the current process's network namespace and into process 1234's namespace. (This is occasionally useful with physical devices, but more useful with virtual devices like veth and macvlan.) CLONE_NEWNS and CLONE_NEWUTS create a deep copy of the current namespace's mount table and hostname/domainname strings, respectively. Further changes in one namespace do not affect the other.