8 ms·

This is getting a little long in the tooth but I created a few videos and did some tests around cgroups back in 2013-2014. Both these help explain the underpinn

This is getting a little long in the tooth but I created a few videos and did some tests around cgroups back in 2013-2014. Both these help explain the underpinnings of Docker and I guess Kubernetes now too. Turtles all the way up. I'm just mentioning this since these are still some of my most popular videos and still the core tech under the hood. A personal note here too. cgroups were invented at Google in the early 2000's. If you're using search today, gmail, docs, maps, etc. You're using cgroups. It sounds simple but this tech really powered a whole wave of innovation/startups/projects that almost everyone interacts with on a daily basis. Either through touching a Google service or interacting with anyone using Docker or Kubernetes (running on cgroups/namespaces). Pretty impressive. #14 - Introduction to Linux Control Groups (Cgroups) https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups https://sysadmincasts.com/episodes/14-introduction-to-linux-... #24 - Introduction to Containers on Linux using LXC https://sysadmincasts.com/episodes/24-introduction-to-containers-on-linux-using-lxc https://sysadmincasts.com/episodes/24-introduction-to-contai...

LeCow 6y ago
Dude, those videos are awesome! Thank you.
caymanjim 6y ago
Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.
- WestCoastJustin 6y ago
  Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years. This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement. Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
  - tyingq 6y ago
    "threw tons of gas on the whole devops movement" I'm happy to find that I agree with you on at least one area of this new ecosystem :)
    - 0xbadcafebee 6y ago
      This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3. The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves) All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months". The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
      - baq 6y ago
        The way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
- kbenson 6y ago
  > If you're running Kubernetes, you're almost always still running Docker. Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
  - Spivak 6y ago
    Podman is meant as a replacement for the docker command as it’s typically used by developers and for bespoke services that can use systemd for lifecycle management. podman is an ergonomic runc wrapper. Redhat’s answer to capital D Docker is cri-o.
    - CameronNemo 6y ago
      To add to this, cri-o and libpod/podman are increasingly sharing code.
- antonvs 6y ago
  > If you're running Kubernetes, you're almost always still running Docker. I'm not sure that's true - a lot of Kubernetes system use containerd these days.
  - jsmith45 6y ago
    I think a lot of documentation still tends to suggest docker as a default, but containerd actually makes more sense. Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers. The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images. The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.
tonetheman 6y ago
Man your site is GREAT. Super clear videos. Good stuff.
contingencies 6y ago
cgroups were invented at Google in the early 2000's. In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time. Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.
- WestCoastJustin 6y ago
  I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google. [1] https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups [2] https://lwn.net/Articles/199643/ https://lwn.net/Articles/199643/
  - contingencies 6y ago
    That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
    - WestCoastJustin 6y ago
      Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
      - contingencies 6y ago
FedericoRazzoli 6y ago
Fantastic videos! Thanks for the links! Yes, Kubernetes uses cgroup too. Systemd units are also based on cgroups.
tofflos 6y ago
These videos were absolutely amazing. Thank you.
obiwanpallav1 6y ago
The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section [1]https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four

LeCow 6y ago

Dude, those videos are awesome! Thank you.

caymanjim 6y ago

Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.

WestCoastJustin 6y ago
Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years. This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement. Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
- tyingq 6y ago
  "threw tons of gas on the whole devops movement" I'm happy to find that I agree with you on at least one area of this new ecosystem :)
  - 0xbadcafebee 6y ago
    This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3. The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves) All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months". The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
    - baq 6y ago
      The way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
kbenson 6y ago
> If you're running Kubernetes, you're almost always still running Docker. Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
- Spivak 6y ago
  Podman is meant as a replacement for the docker command as it’s typically used by developers and for bespoke services that can use systemd for lifecycle management. podman is an ergonomic runc wrapper. Redhat’s answer to capital D Docker is cri-o.
  - CameronNemo 6y ago
    To add to this, cri-o and libpod/podman are increasingly sharing code.
antonvs 6y ago
> If you're running Kubernetes, you're almost always still running Docker. I'm not sure that's true - a lot of Kubernetes system use containerd these days.
- jsmith45 6y ago
  I think a lot of documentation still tends to suggest docker as a default, but containerd actually makes more sense. Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers. The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images. The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.

tonetheman 6y ago

Man your site is GREAT. Super clear videos. Good stuff.

contingencies 6y ago

cgroups were invented at Google in the early 2000's. In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time. Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.

WestCoastJustin 6y ago
I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google. [1] https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups [2] https://lwn.net/Articles/199643/ https://lwn.net/Articles/199643/
- contingencies 6y ago
  That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
  - WestCoastJustin 6y ago
    Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
    - contingencies 6y ago

FedericoRazzoli 6y ago

Fantastic videos! Thanks for the links! Yes, Kubernetes uses cgroup too. Systemd units are also based on cgroups.

tofflos 6y ago

These videos were absolutely amazing. Thank you.

obiwanpallav1 6y ago

The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section [1]https://www.redhat.com/sysadmin/cgroups-part-four https://www.redhat.com/sysadmin/cgroups-part-four