5 ms·

Tapes and cards!! OMG that puts JCL into perspective! Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say D

by tombh 5y ago

Tapes and cards!! OMG that puts JCL into perspective! Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say Docker was eliminated from Kubernetes, it was simply made optional. I think it's hard to argue that Docker isn't still the pioneer, like I mentioned Docker is the OG market fit which all the other runtimes are trying to emulate. So well-designed APIs, which Mesos doesn't have, I can believe that. But I think it's fair to say Flynn and Deis put a lot of effort into theirs. I forgot about Rancher and Fleet! I don't know about their APIs. For sure Flynn and Deis never got as far as CRDs, but they still achieved those basic concepts we use in Kubernetes today; deployments, storage, config, logs, routes, networks, etc. I'm just still not convinced that Borg did anything new or innovative that wasn't already in some Open Source project before Kubernetes came along.

scott_s 5y ago
The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/
p_l 5y ago
Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s). Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images. As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS). A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-). The implementation details of them aren't for end user consumption, and generally the less they leak the better. OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs. Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources. Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images. Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails). Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore). Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details) 2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer". Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?) Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container. By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly. Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development. And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.
- pjmlp 5y ago
  Great overview, quite interesting. Just to add a bit more of history, around 2000, HP starts pushing for HP-UX Vaults as means to also package application into production and restrict what resources they can use from the OS.
  - p_l 5y ago
    Do you have some more links? I did encounter "Vault" themed name in relation to HP-UX, but it sounded more like Trusted Solaris or similar MAC/MLS enabled unix version - but then I had literally a sentence or two for description.
    - pjmlp 5y ago
      Unfortunately not much much, given how HPE now kind of killed all the nice documents that HP-UX used to have 20 years ago. This is what I can still find, https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c02791255 https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
      - p_l 5y ago
        Looks like something closer to Solaris Zones (or maybe AIX WPAR?), definitely interesting addition to resource management building block of the journey!
- tombh 5y ago
  This is amazing! Thanks so much for writing it out. It's so good to read about the same story from a different perspective. I'm only going to respond to one theme in it, I hope that doesn't come across as me not appreciating or not agreeing with what you wrote. Docker's core innovation from a _technical_ perspective may well be packaging, but I think that really misrepresents the history. Docker, like Heroku, made ordinary developers powerful. It did that through _culture_, by speaking the right language, by knowing what developers want and what they're capable of doing. I want to talk about Jeff Lindsay[1], the co-author of Docker. He's a prolific open source contributor, a vocal critic of VC funding (and I suspect HN) and passionate advocate of the basic building blocks of software. He created Dokku, introduced Alpine Linux to Docker, founded Flynn, and Glider Labs which innovated things like service discovery and distributed logging in Docker. Jeff lives and breathes hacker culture. His hallmark is written all over Docker: embrace open source, provide basic building blocks and encourage innovation. He's one of us, a hacker, he builds things for the sheer joy of it. It's obvious he's not in it for the big enterprise money. I mean he's not even it for the reputation, hardly anybody knows his name, everyone just thinks Solomon Hykes created Docker. These are the people that need to be mentioned in the story, in that documentary. The people that put things into the world, because the world needed them. I don't know the name of the person that came up with The 12 Factors, so it's the same thing, someone knew that the world needed them and just gave them to us. Of course the story of Omega and Borg is crucial, but they were always motivated by the private interests of an unimaginably large corporation, a corporation that removed its own "Do No Evil" aspiration. Jeff Lindsay has a power that Google doesn't have, the love of software for software's sake. That is a critical part of why we have Docker, and why it became so popular. And Docker's open source status and developer community is an inescapable basis for the world which Kubernetes now rules over. 1. https://dev.to/progrium/hi-im-jeff-lindsay https://dev.to/progrium/hi-im-jeff-lindsay

by tombh 5y ago

scott_s 5y ago

The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/

p_l 5y ago

Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s). Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images. As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS). A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-). The implementation details of them aren't for end user consumption, and generally the less they leak the better. OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs. Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources. Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images. Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails). Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore). Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details) 2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer". Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?) Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container. By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly. Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development. And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.

pjmlp 5y ago
Great overview, quite interesting. Just to add a bit more of history, around 2000, HP starts pushing for HP-UX Vaults as means to also package application into production and restrict what resources they can use from the OS.
- p_l 5y ago
  Do you have some more links? I did encounter "Vault" themed name in relation to HP-UX, but it sounded more like Trusted Solaris or similar MAC/MLS enabled unix version - but then I had literally a sentence or two for description.
  - pjmlp 5y ago
    Unfortunately not much much, given how HPE now kind of killed all the nice documents that HP-UX used to have 20 years ago. This is what I can still find, https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c02791255 https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
    - p_l 5y ago
      Looks like something closer to Solaris Zones (or maybe AIX WPAR?), definitely interesting addition to resource management building block of the journey!
tombh 5y ago
This is amazing! Thanks so much for writing it out. It's so good to read about the same story from a different perspective. I'm only going to respond to one theme in it, I hope that doesn't come across as me not appreciating or not agreeing with what you wrote. Docker's core innovation from a _technical_ perspective may well be packaging, but I think that really misrepresents the history. Docker, like Heroku, made ordinary developers powerful. It did that through _culture_, by speaking the right language, by knowing what developers want and what they're capable of doing. I want to talk about Jeff Lindsay[1], the co-author of Docker. He's a prolific open source contributor, a vocal critic of VC funding (and I suspect HN) and passionate advocate of the basic building blocks of software. He created Dokku, introduced Alpine Linux to Docker, founded Flynn, and Glider Labs which innovated things like service discovery and distributed logging in Docker. Jeff lives and breathes hacker culture. His hallmark is written all over Docker: embrace open source, provide basic building blocks and encourage innovation. He's one of us, a hacker, he builds things for the sheer joy of it. It's obvious he's not in it for the big enterprise money. I mean he's not even it for the reputation, hardly anybody knows his name, everyone just thinks Solomon Hykes created Docker. These are the people that need to be mentioned in the story, in that documentary. The people that put things into the world, because the world needed them. I don't know the name of the person that came up with The 12 Factors, so it's the same thing, someone knew that the world needed them and just gave them to us. Of course the story of Omega and Borg is crucial, but they were always motivated by the private interests of an unimaginably large corporation, a corporation that removed its own "Do No Evil" aspiration. Jeff Lindsay has a power that Google doesn't have, the love of software for software's sake. That is a critical part of why we have Docker, and why it became so popular. And Docker's open source status and developer community is an inescapable basis for the world which Kubernetes now rules over. 1. https://dev.to/progrium/hi-im-jeff-lindsay https://dev.to/progrium/hi-im-jeff-lindsay