5 ms·

I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [

I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google. [1] https://en.wikipedia.org/wiki/Cgroups https://en.wikipedia.org/wiki/Cgroups [2] https://lwn.net/Articles/199643/ https://lwn.net/Articles/199643/

contingencies 6y ago
That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
- WestCoastJustin 6y ago
  Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
  - contingencies 6y ago
    I am pretty sure that OpenVZ also predates the Google work in a Linux context. Basically saying that 'containers on Linux' are 'made at Google' is not true.
    - deleted 6y ago
      [deleted]
    - ceph_ 6y ago
      But the OP doesn't state containers on linux were invented at Google. It says cgroups were.
    - deleted 6y ago
      [deleted]
  - _jal 6y ago
    BSD jails date back to FreeBSD 4, which was released in 2000. Edit: I started questioning my memory of this, so I poked around some. Poul-Henning Kamp did the initial work in April, 1999, here is the commit: https://svnweb.freebsd.org/base?view=revision&revision=46155 https://svnweb.freebsd.org/base?view=revision&revision=46155 More details from him about it: http://phk.freebsd.dk/sagas/jails/ http://phk.freebsd.dk/sagas/jails/
    - WestCoastJustin 6y ago
      Awesome, thanks _jal!
  - antb123 6y ago
    Mid 90s ran some vm system of MSDOS. It added threading and was for phone systems.
- nix23 6y ago
  But if your talking about system virtualization, System 370 invented that way before FreeBSD invented jails...and speaking of jails..even chroot on BSD came before that, but he is talking about cgroups.
  - msla 6y ago
    If you're talking about what I think you're talking about, IBM's VM is a hypervisor, which is a world away from cgroups, chroot, jails, or, say, the Java VM, before anyone brings that up.
    - nix23 6y ago
      I talk about the CP/CMS Operating-system, CP-370 later VM/370 "virtualized or containerized" just the OS and not Hardware (for accounting and Security reasons): https://en.wikipedia.org/wiki/CP/CMS#Overview https://en.wikipedia.org/wiki/CP/CMS#Overview that's exactly what jails/zones do (System, NOT Hardware virtualization)
      - deleted 6y ago
        [deleted]
      - msla 6y ago
        No, it isn't, and you don't understand one or the other or both.
- ar-jan 6y ago
  There's also Linux Vserver (2001), which used to be included in Debian, but is still in use today.
- bogomipz 6y ago
  Indeed from the OpenVZ wiki's history section: >"Nov 1999: Alexander Tormasov visited Singapore and proposed a new direction to Sergey Beloussov: container virtualization. He formulated three main components: containers as a set of processes with namespace isolation, file system to share code/ram and isolation in resources. Indeed it was 1999 when our engineers started adding bits and pieces of containers technology to Linux kernel 2.2. Well, not exactly "containers", but rather "virtual environments" at that time -- as it often happens with new technologies, the terminology was different (the term "container" was coined by Sun only five years later, in 2004)."[1] [1] https://wiki.openvz.org/History https://wiki.openvz.org/History
- pjmlp 6y ago
  And HP-UX Vaults (I used them in 2000) and Tru64 containers as well.
- menage 6y ago
  It's probably true that public use of such technology can be dated to LXC. But cgroups, and the initial (private) userspace implementations, definitely come from Google. There were many efforts to get some kind of container technology into Linux, from the early 2000s (and probably earlier). VServer, OpenVZ, etc, were all trying to support full server virtualization. Cgroups was different from most of them in that it wasn't trying to present a virtual server environment, just facilitate resource control. At Google, all the jobs we ran on Borg (the internal system that later inspired Kubernetes) knew that they were in a shared environment, so there was much less need to fake a private virtual environment. The low-level libraries linked into all Google binaries did a lot of coordination with Borg to make the sharing fairly painless. But we were trying to run tens of jobs on each of our increasingly-larger machines, and some of the batch jobs would inevitably end up hogging resources, which would hurt the important latency-sensitive jobs. CPU and memory isolation were the main things that we cared about, with network/disk as secondary elements. For a while (in 2005-2007) in Borg we approximated this by using the kernel's "fake NUMA" support (originally intended for testing the NUMA code) to break up the system into a bunch of fake NUMA nodes, and using cpusets to reserve CPUs and memory chunks to important jobs. It was pretty ugly and rather coarse-grained, but it worked rather well and was the first widespread (running on millions of Google Linux servers) userspace system for controlling a cgroups-like system. Essentially, cgroups piggy-backed on to the existing cpusets mechanism/API which had already been accepted into the kernel, and expanded it into a more generic way of creating a hierarchy of groups and mapping a process to a group. This was a lot simpler to get accepted than an entire virtualization system. Given that mapping, making other resource scheduling be group-aware rather than just process-aware was much more straightforward. The same userspace support in Borg that had been used for controlling cpusets worked pretty much as-is with cgroups, since the basic API was the same (just more resources were supported).
- musicale 6y ago
  Solaris Zones is another Unix container system, ~2004-2005.
- musicale 6y ago
  It's important to realize that Linux offers a-la-carte container features which evolved over time. Some of the features include chroot, process groups, user/group permissions, cgroups, network multiplexing (e.g. inetd, nat), namespaces (network, mount, pid, user, etc.) and unshare/setns, union file systems like unionfs/aufs/overlayfs, etc.. There are a lot of ways you can implement container orchestration systems on Linux. And as noted, OpenVZ and Linux Vserver were earlier container subsystems for Linux.
fulafel 6y ago
Intersting Linux history - Do you by a chance have a link to the commit ? I picked a file from the patch index mail linked above, mm_inline.h, and went scrolling through its history at https://github.com/torvalds/linux/commits/master/include/linux/mm_inline.h https://github.com/torvalds/linux/commits/master/include/lin... but didn't see a corresponding change there. I guess the patches might have gotten refactored before merging or something too but would be nice to have a pointer in the Linux tree history that would work as a reference. edit: tried looking for linux/container.h too but that just brings up a newer acpi container related container.h: https://github.com/torvalds/linux/commits/master/include/linux/container.h https://github.com/torvalds/linux/commits/master/include/lin... edit 2: history for linux/cgroup.c starts in a year later in 2007, with this commit: https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c01cac3dec71f118cd8ac2 https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c... - it has mentions of people from OpenVZ, IBM and a bunch of unaffiliated domains in addition to the signed-off-by line from a Googler (Paul Menage).
- merb 6y ago
  I think the patch never made it trough mainline and the work on the cgroups framework started. with this patch. It basically says: Based originally on the cpuset system, extracted by Paul Menage * Copyright (C) 2006 Google, Inc * Copyright notices from the original cpuset code: * -------------------------------------------------- * Copyright (C) 2003 BULL SA. * Copyright (C) 2004-2006 Silicon Graphics, Inc. * * Portions derived from Patrick Mochel's sysfs code. * sysfs is Copyright (c) 2001-3 Patrick Mochel * * 2003-10-10 Written by Simon Derr. * 2003-10-22 Updates by Stephen Hemminger. * 2004 May-July Rework by Paul Jackson. just wow how old this is.
bogomipz 6y ago
I believe this is the original whitepaper from 2007. "Adding Generic Process Containers to the Linux Kernel" https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pdf https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pd... Although the paper was written by Paul B. Menage it looks like Balbir Singh and Srivatsa Vaddagiri from IBM also made contributions to it.

contingencies 6y ago

That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.

WestCoastJustin 6y ago
Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
- contingencies 6y ago
  I am pretty sure that OpenVZ also predates the Google work in a Linux context. Basically saying that 'containers on Linux' are 'made at Google' is not true.
  - deleted 6y ago
    [deleted]
  - ceph_ 6y ago
    But the OP doesn't state containers on linux were invented at Google. It says cgroups were.
  - deleted 6y ago
    [deleted]
- _jal 6y ago
  BSD jails date back to FreeBSD 4, which was released in 2000. Edit: I started questioning my memory of this, so I poked around some. Poul-Henning Kamp did the initial work in April, 1999, here is the commit: https://svnweb.freebsd.org/base?view=revision&revision=46155 https://svnweb.freebsd.org/base?view=revision&revision=46155 More details from him about it: http://phk.freebsd.dk/sagas/jails/ http://phk.freebsd.dk/sagas/jails/
  - WestCoastJustin 6y ago
    Awesome, thanks _jal!
- antb123 6y ago
  Mid 90s ran some vm system of MSDOS. It added threading and was for phone systems.
nix23 6y ago
But if your talking about system virtualization, System 370 invented that way before FreeBSD invented jails...and speaking of jails..even chroot on BSD came before that, but he is talking about cgroups.
- msla 6y ago
  If you're talking about what I think you're talking about, IBM's VM is a hypervisor, which is a world away from cgroups, chroot, jails, or, say, the Java VM, before anyone brings that up.
  - nix23 6y ago
    I talk about the CP/CMS Operating-system, CP-370 later VM/370 "virtualized or containerized" just the OS and not Hardware (for accounting and Security reasons): https://en.wikipedia.org/wiki/CP/CMS#Overview https://en.wikipedia.org/wiki/CP/CMS#Overview that's exactly what jails/zones do (System, NOT Hardware virtualization)
    - deleted 6y ago
      [deleted]
    - msla 6y ago
      No, it isn't, and you don't understand one or the other or both.
ar-jan 6y ago
There's also Linux Vserver (2001), which used to be included in Debian, but is still in use today.
bogomipz 6y ago
Indeed from the OpenVZ wiki's history section: >"Nov 1999: Alexander Tormasov visited Singapore and proposed a new direction to Sergey Beloussov: container virtualization. He formulated three main components: containers as a set of processes with namespace isolation, file system to share code/ram and isolation in resources. Indeed it was 1999 when our engineers started adding bits and pieces of containers technology to Linux kernel 2.2. Well, not exactly "containers", but rather "virtual environments" at that time -- as it often happens with new technologies, the terminology was different (the term "container" was coined by Sun only five years later, in 2004)."[1] [1] https://wiki.openvz.org/History https://wiki.openvz.org/History
pjmlp 6y ago
And HP-UX Vaults (I used them in 2000) and Tru64 containers as well.
menage 6y ago
It's probably true that public use of such technology can be dated to LXC. But cgroups, and the initial (private) userspace implementations, definitely come from Google. There were many efforts to get some kind of container technology into Linux, from the early 2000s (and probably earlier). VServer, OpenVZ, etc, were all trying to support full server virtualization. Cgroups was different from most of them in that it wasn't trying to present a virtual server environment, just facilitate resource control. At Google, all the jobs we ran on Borg (the internal system that later inspired Kubernetes) knew that they were in a shared environment, so there was much less need to fake a private virtual environment. The low-level libraries linked into all Google binaries did a lot of coordination with Borg to make the sharing fairly painless. But we were trying to run tens of jobs on each of our increasingly-larger machines, and some of the batch jobs would inevitably end up hogging resources, which would hurt the important latency-sensitive jobs. CPU and memory isolation were the main things that we cared about, with network/disk as secondary elements. For a while (in 2005-2007) in Borg we approximated this by using the kernel's "fake NUMA" support (originally intended for testing the NUMA code) to break up the system into a bunch of fake NUMA nodes, and using cpusets to reserve CPUs and memory chunks to important jobs. It was pretty ugly and rather coarse-grained, but it worked rather well and was the first widespread (running on millions of Google Linux servers) userspace system for controlling a cgroups-like system. Essentially, cgroups piggy-backed on to the existing cpusets mechanism/API which had already been accepted into the kernel, and expanded it into a more generic way of creating a hierarchy of groups and mapping a process to a group. This was a lot simpler to get accepted than an entire virtualization system. Given that mapping, making other resource scheduling be group-aware rather than just process-aware was much more straightforward. The same userspace support in Borg that had been used for controlling cpusets worked pretty much as-is with cgroups, since the basic API was the same (just more resources were supported).
musicale 6y ago
Solaris Zones is another Unix container system, ~2004-2005.
musicale 6y ago
It's important to realize that Linux offers a-la-carte container features which evolved over time. Some of the features include chroot, process groups, user/group permissions, cgroups, network multiplexing (e.g. inetd, nat), namespaces (network, mount, pid, user, etc.) and unshare/setns, union file systems like unionfs/aufs/overlayfs, etc.. There are a lot of ways you can implement container orchestration systems on Linux. And as noted, OpenVZ and Linux Vserver were earlier container subsystems for Linux.

fulafel 6y ago

Intersting Linux history - Do you by a chance have a link to the commit ? I picked a file from the patch index mail linked above, mm_inline.h, and went scrolling through its history at https://github.com/torvalds/linux/commits/master/include/linux/mm_inline.h https://github.com/torvalds/linux/commits/master/include/lin... but didn't see a corresponding change there. I guess the patches might have gotten refactored before merging or something too but would be nice to have a pointer in the Linux tree history that would work as a reference. edit: tried looking for linux/container.h too but that just brings up a newer acpi container related container.h: https://github.com/torvalds/linux/commits/master/include/linux/container.h https://github.com/torvalds/linux/commits/master/include/lin... edit 2: history for linux/cgroup.c starts in a year later in 2007, with this commit: https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c01cac3dec71f118cd8ac2 https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c... - it has mentions of people from OpenVZ, IBM and a bunch of unaffiliated domains in addition to the signed-off-by line from a Googler (Paul Menage).

merb 6y ago
I think the patch never made it trough mainline and the work on the cgroups framework started. with this patch. It basically says: Based originally on the cpuset system, extracted by Paul Menage * Copyright (C) 2006 Google, Inc * Copyright notices from the original cpuset code: * -------------------------------------------------- * Copyright (C) 2003 BULL SA. * Copyright (C) 2004-2006 Silicon Graphics, Inc. * * Portions derived from Patrick Mochel's sysfs code. * sysfs is Copyright (c) 2001-3 Patrick Mochel * * 2003-10-10 Written by Simon Derr. * 2003-10-22 Updates by Stephen Hemminger. * 2004 May-July Rework by Paul Jackson. just wow how old this is.

bogomipz 6y ago

I believe this is the original whitepaper from 2007. "Adding Generic Process Containers to the Linux Kernel" https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pdf https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pd... Although the paper was written by Paul B. Menage it looks like Balbir Singh and Srivatsa Vaddagiri from IBM also made contributions to it.