9 ms·

A Linux sysadmin's introduction to cgroups

WestCoastJustin 6y ago
This is getting a little long in the tooth but I created a few videos and did some tests around cgroups back in 2013-2014. Both these help explain the underpinnings of Docker and I guess Kubernetes now too. Turtles all the way up. I'm just mentioning this since these are still some of my most popular videos and still the core tech under the hood. A personal note here too. cgroups were invented at Google in the early 2000's. If you're using search today, gmail, docs, maps, etc. You're using cgroups. It sounds simple but this tech really powered a whole wave of innovation/startups/projects that almost everyone interacts with on a daily basis. Either through touching a Google service or interacting with anyone using Docker or Kubernetes (running on cgroups/namespaces). Pretty impressive. #14 - Introduction to Linux Control Groups (Cgroups) https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups https://sysadmincasts.com/episodes/14-introduction-to-linux-... #24 - Introduction to Containers on Linux using LXC https://sysadmincasts.com/episodes/24-introduction-to-containers-on-linux-using-lxc https://sysadmincasts.com/episodes/24-introduction-to-contai...
- LeCow 6y ago
  Dude, those videos are awesome! Thank you.
- caymanjim 6y ago
  Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.
  - WestCoastJustin 6y ago
    Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years. This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement. Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
    - tyingq 6y ago
      "threw tons of gas on the whole devops movement" I'm happy to find that I agree with you on at least one area of this new ecosystem :)
      - 0xbadcafebee 6y ago
        This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3. The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves) All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months". The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
        baq 6y ago
        The way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
  - kbenson 6y ago
    > If you're running Kubernetes, you're almost always still running Docker. Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
    - Spivak 6y ago
      Podman is meant as a replacement for the docker command as it’s typically used by developers and for bespoke services that can use systemd for lifecycle management. podman is an ergonomic runc wrapper. Redhat’s answer to capital D Docker is cri-o.
      - CameronNemo 6y ago
        To add to this, cri-o and libpod/podman are increasingly sharing code.
  - antonvs 6y ago
    > If you're running Kubernetes, you're almost always still running Docker. I'm not sure that's true - a lot of Kubernetes system use containerd these days.
    - jsmith45 6y ago
      I think a lot of documentation still tends to suggest docker as a default, but containerd actually makes more sense. Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers. The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images. The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.
- tonetheman 6y ago
  Man your site is GREAT. Super clear videos. Good stuff.
- contingencies 6y ago
  cgroups were invented at Google in the early 2000's. In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time. Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.
  - WestCoastJustin 6y ago
    I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google. [1] https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups [2] https://lwn.net/Articles/199643/ https://lwn.net/Articles/199643/
    - contingencies 6y ago
      That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
      - WestCoastJustin 6y ago
        Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
        contingencies 6y ago
- FedericoRazzoli 6y ago
  Fantastic videos! Thanks for the links! Yes, Kubernetes uses cgroup too. Systemd units are also based on cgroups.
- tofflos 6y ago
  These videos were absolutely amazing. Thank you.
- obiwanpallav1 6y ago
  The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section [1]https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four
deleted 6y ago
[deleted]
jeffbee 6y ago
These things should strive for accuracy and this article is not accurate. The cgroups facility does not control "the number of CPU shares per process." Although you can put such a thing into effect with control groups, it's more accurate to say that a control group limits the resources of a set of tasks. Those tasks may be from one or several processes, and it's also the case that a single process can divide its own tasks into several cgroups.
- waynesonfire 6y ago
  Hey webster, whats a task?
- stuff4ben 6y ago
  While I agree that we should strive for accuracy, you're really splitting hairs there. Plus the author puts a disclaimer in there "NOTE: This is a gross simplification and is NOT technically accurate should you be looking to get involved with the underlying cgroup code. The above explanation is meant for clarity of understanding."
  - jeffbee 6y ago
    But the explanation does not contribute to clarity of understanding. This has nothing to do with hacking on cgroups itself. As a user of cgroups you should know that one group may control multiple processes, and one process may be controlled by multiple groups! There certainly is not a 1:1 correspondence.
  - kortilla 6y ago
    It’s not an internal detail, it’s fundamental to understanding it from a user perspective.
flexops 6y ago
Found an interesting ongoing blogpost series that also tries to explain all the other low level kernel mechanism that make up Docker and other container technologies on Linux. They haven't reach the topic of control groups though. https://www.schutzwerk.com/en/43/posts/linux_container_intro/ https://www.schutzwerk.com/en/43/posts/linux_container_intro...
- FedericoRazzoli 6y ago
  Still work in progress and the topics are not too in-depth... but it's interesting, thanks.
ape4 6y ago
systemd uses cgroups so nearly every Linux box is using them.
craigsmansion 6y ago
Ah yes, "cgroups", according to notable no-nonsense kernel hacker Al Viro: "it's not just badly written kernel/cgroup.c - the interfaces on both sides (userland and the rest of kernel) are seriously misdesigned. As far as I'm concerned, configuring it out solves my problem nicely." That was in 2011, so things might have improved. What remains however is that cgroups was added to the kernel, by Googlers, for easier maintenance, but with an implicit understanding that no sane person would actually make use of it to do something important. ... enter SystemD.
- jeffbee 6y ago
  The whole google is built on cgroups v1, which seems to stand as an existence proof that these interfaces are not useless.
  - yjftsjthsd-h 6y ago
    "bad design" and "useful" can be overlapping groups
  - PaulDavisThe1st 6y ago
    so when someone is using gmail on a macOS system, how are they using cgroups in any way? how is "the whole google" built on this? perhaps you means "google's web-facing servers using cgroups extensively" ?
    - jeffbee 6y ago
      That is a somewhat weird and dare I say pointless take. Thousands of machines are involved in serving every request from a web browser to gmail. Of those, only the web browser might not be in linux cgroups.
- menage 6y ago
  It's true that a better userspace API for cgroups might have involved something more like ioctl(), rather than a filesystem with text-based control files. But people had been trying to get such APIs into the kernel for years with no success. Which is why we tried to piggyback on something existing. Cpusets, which had already been accepted into the kernel a year or two previously, provided the basic userspace API for cgroups. It was just expanded to support control files for more resource types. (And multiple resource hierarchies, although that's gone in cgroups v2 I think).
dianhuji1 6y ago
so great info
grantseltzer 6y ago
I wrote a blog post on cgroups a couple years ago that's still accurate and goes further into depth and gives workable examples of using them both inside and outside of containers: https://www.grant.pizza/blog/understanding-cgroups/ https://www.grant.pizza/blog/understanding-cgroups/
aschatten 6y ago
This is Part I from a series. Part II: https://www.redhat.com/sysadmin/cgroups-part-two https://www.redhat.com/sysadmin/cgroups-part-two Part III: https://www.redhat.com/sysadmin/cgroups-part-three https://www.redhat.com/sysadmin/cgroups-part-three Part IV: https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four
- ldng 6y ago
  Someone managed to apply part three ? Systemd gets in the way for me. Edit: well, at least part four, finally, properly document how to cope with systemd. And guess what, it is different than 90% of documentation on cgroups on the internet.
meatspaceping 6y ago
Conspiracy theory. our kubernetes clusters as a service are not really running on VMs.
musicale 6y ago
I like the cgroups approach because it seems to logically build on the established idea of process groups. I also tend to think that processes in cgroups are a sweet spot of lightweight containerization that can do quite a bit.

This is getting a little long in the tooth but I created a few videos and did some tests around cgroups back in 2013-2014. Both these help explain the underpinnings of Docker and I guess Kubernetes now too. Turtles all the way up. I'm just mentioning this since these are still some of my most popular videos and still the core tech under the hood. A personal note here too. cgroups were invented at Google in the early 2000's. If you're using search today, gmail, docs, maps, etc. You're using cgroups. It sounds simple but this tech really powered a whole wave of innovation/startups/projects that almost everyone interacts with on a daily basis. Either through touching a Google service or interacting with anyone using Docker or Kubernetes (running on cgroups/namespaces). Pretty impressive. #14 - Introduction to Linux Control Groups (Cgroups) https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups https://sysadmincasts.com/episodes/14-introduction-to-linux-... #24 - Introduction to Containers on Linux using LXC https://sysadmincasts.com/episodes/24-introduction-to-containers-on-linux-using-lxc https://sysadmincasts.com/episodes/24-introduction-to-contai...

LeCow 6y ago
Dude, those videos are awesome! Thank you.
caymanjim 6y ago
Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.
- WestCoastJustin 6y ago
  Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years. This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement. Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
  - tyingq 6y ago
    "threw tons of gas on the whole devops movement" I'm happy to find that I agree with you on at least one area of this new ecosystem :)
    - 0xbadcafebee 6y ago
      This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3. The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves) All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months". The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
      - baq 6y ago
        The way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
- kbenson 6y ago
  > If you're running Kubernetes, you're almost always still running Docker. Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
  - Spivak 6y ago
    Podman is meant as a replacement for the docker command as it’s typically used by developers and for bespoke services that can use systemd for lifecycle management. podman is an ergonomic runc wrapper. Redhat’s answer to capital D Docker is cri-o.
    - CameronNemo 6y ago
      To add to this, cri-o and libpod/podman are increasingly sharing code.
- antonvs 6y ago
  > If you're running Kubernetes, you're almost always still running Docker. I'm not sure that's true - a lot of Kubernetes system use containerd these days.
  - jsmith45 6y ago
    I think a lot of documentation still tends to suggest docker as a default, but containerd actually makes more sense. Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers. The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images. The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.
tonetheman 6y ago
Man your site is GREAT. Super clear videos. Good stuff.
contingencies 6y ago
cgroups were invented at Google in the early 2000's. In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time. Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.
- WestCoastJustin 6y ago
  I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google. [1] https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups [2] https://lwn.net/Articles/199643/ https://lwn.net/Articles/199643/
  - contingencies 6y ago
    That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
    - WestCoastJustin 6y ago
      Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
      - contingencies 6y ago
FedericoRazzoli 6y ago
Fantastic videos! Thanks for the links! Yes, Kubernetes uses cgroup too. Systemd units are also based on cgroups.
tofflos 6y ago
These videos were absolutely amazing. Thank you.
obiwanpallav1 6y ago
The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section [1]https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four

deleted 6y ago

[deleted]

jeffbee 6y ago

These things should strive for accuracy and this article is not accurate. The cgroups facility does not control "the number of CPU shares per process." Although you can put such a thing into effect with control groups, it's more accurate to say that a control group limits the resources of a set of tasks. Those tasks may be from one or several processes, and it's also the case that a single process can divide its own tasks into several cgroups.

waynesonfire 6y ago
Hey webster, whats a task?
stuff4ben 6y ago
While I agree that we should strive for accuracy, you're really splitting hairs there. Plus the author puts a disclaimer in there "NOTE: This is a gross simplification and is NOT technically accurate should you be looking to get involved with the underlying cgroup code. The above explanation is meant for clarity of understanding."
- jeffbee 6y ago
  But the explanation does not contribute to clarity of understanding. This has nothing to do with hacking on cgroups itself. As a user of cgroups you should know that one group may control multiple processes, and one process may be controlled by multiple groups! There certainly is not a 1:1 correspondence.
- kortilla 6y ago
  It’s not an internal detail, it’s fundamental to understanding it from a user perspective.

flexops 6y ago

Found an interesting ongoing blogpost series that also tries to explain all the other low level kernel mechanism that make up Docker and other container technologies on Linux. They haven't reach the topic of control groups though. https://www.schutzwerk.com/en/43/posts/linux_container_intro/ https://www.schutzwerk.com/en/43/posts/linux_container_intro...

FedericoRazzoli 6y ago
Still work in progress and the topics are not too in-depth... but it's interesting, thanks.

ape4 6y ago

systemd uses cgroups so nearly every Linux box is using them.

craigsmansion 6y ago

Ah yes, "cgroups", according to notable no-nonsense kernel hacker Al Viro: "it's not just badly written kernel/cgroup.c - the interfaces on both sides (userland and the rest of kernel) are seriously misdesigned. As far as I'm concerned, configuring it out solves my problem nicely." That was in 2011, so things might have improved. What remains however is that cgroups was added to the kernel, by Googlers, for easier maintenance, but with an implicit understanding that no sane person would actually make use of it to do something important. ... enter SystemD.

jeffbee 6y ago
The whole google is built on cgroups v1, which seems to stand as an existence proof that these interfaces are not useless.
- yjftsjthsd-h 6y ago
  "bad design" and "useful" can be overlapping groups
- PaulDavisThe1st 6y ago
  so when someone is using gmail on a macOS system, how are they using cgroups in any way? how is "the whole google" built on this? perhaps you means "google's web-facing servers using cgroups extensively" ?
  - jeffbee 6y ago
    That is a somewhat weird and dare I say pointless take. Thousands of machines are involved in serving every request from a web browser to gmail. Of those, only the web browser might not be in linux cgroups.
menage 6y ago
It's true that a better userspace API for cgroups might have involved something more like ioctl(), rather than a filesystem with text-based control files. But people had been trying to get such APIs into the kernel for years with no success. Which is why we tried to piggyback on something existing. Cpusets, which had already been accepted into the kernel a year or two previously, provided the basic userspace API for cgroups. It was just expanded to support control files for more resource types. (And multiple resource hierarchies, although that's gone in cgroups v2 I think).

dianhuji1 6y ago

so great info

grantseltzer 6y ago

I wrote a blog post on cgroups a couple years ago that's still accurate and goes further into depth and gives workable examples of using them both inside and outside of containers: https://www.grant.pizza/blog/understanding-cgroups/ https://www.grant.pizza/blog/understanding-cgroups/

aschatten 6y ago

This is Part I from a series. Part II: https://www.redhat.com/sysadmin/cgroups-part-two https://www.redhat.com/sysadmin/cgroups-part-two Part III: https://www.redhat.com/sysadmin/cgroups-part-three https://www.redhat.com/sysadmin/cgroups-part-three Part IV: https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four

ldng 6y ago
Someone managed to apply part three ? Systemd gets in the way for me. Edit: well, at least part four, finally, properly document how to cope with systemd. And guess what, it is different than 90% of documentation on cgroups on the internet.

meatspaceping 6y ago

Conspiracy theory. our kubernetes clusters as a service are not really running on VMs.

musicale 6y ago

I like the cgroups approach because it seems to logically build on the established idea of process groups. I also tend to think that processes in cgroups are a sweet spot of lightweight containerization that can do quite a bit.