8 ms·

Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look. I totally know what you mean about Ope

• p_l 5y ago
  JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved. Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system). Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem. As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny. Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed. The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API). A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)
  • tombh 5y ago
    Tapes and cards!! OMG that puts JCL into perspective! Sure Kubernetes' API hid Docker, but its marketing certainly didn't. And I'm not sure it's right to say Docker was eliminated from Kubernetes, it was simply made optional. I think it's hard to argue that Docker isn't still the pioneer, like I mentioned Docker is the OG market fit which all the other runtimes are trying to emulate. So well-designed APIs, which Mesos doesn't have, I can believe that. But I think it's fair to say Flynn and Deis put a lot of effort into theirs. I forgot about Rancher and Fleet! I don't know about their APIs. For sure Flynn and Deis never got as far as CRDs, but they still achieved those basic concepts we use in Kubernetes today; deployments, storage, config, logs, routes, networks, etc. I'm just still not convinced that Borg did anything new or innovative that wasn't already in some Open Source project before Kubernetes came along.
    • scott_s 5y ago
      The EuroSys paper on Borg is a good place to get the background on Borg, how it influenced Kube, and how it relates to other systems: https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/
    • p_l 5y ago
      Well, the issue is that Borg is not public, and isn't direct ancestor of kubernetes - a bunch of design decisions were informed by the Omega project which was an attempt to build a new Borg (for example, pluggable schedulers, a rather core aspect of k8s). Marketing hanging onto Docker was a great marketing move, even if all that docker brought was essentially images. As for basic concepts - I believe that the core issue is that Flynn/Deis/Heroku converge from the opposite starting point (it's arguably a common case for all PaaS). A PaaS wants to provide those high-level building blocks to the end user, but (especially the case with Heroku and other closed source examples), it hides everything else. In a way it is an offshoot of, if much better designed usually, of the classic LAMP experience, except usually with some extras provided - and these days we get APIs instead of crappy web panel (or cPanel, in fact ^_-). The implementation details of them aren't for end user consumption, and generally the less they leak the better. OTOH, we have infrastucture-side systems, which start with "I have X resources and I want to utilise them efficiently". Those tend to overlap with PaaS in the sense that often you need them to implement a PaaS, but how generic they are differs. Old MPP schedulers for supercomputer clusters and things like VMScluster generic batch queue are probably some of the most obvious intermediate systems between IBM ASP and 21st century. Some proprietary, some open source just niche. They didn't necessarily handle all those higher level ideaas like deployments (closer to basic k8s ReplicationController), often disregarded storage (single system image, cluster-wide common shared mounts, cluster-wide availability of storage in VMScluster, etc.). At the same time High-Availability focused clustering was also building up, but usually with statically allocated resources. Generally those two branches of clustering started to combine around 2000 (possibly earlier - I am limited here by my own knowledge), with "grid computing" combining ideas of supercomputing scheduling, opportunistic clustering and reliability. It's also around that time that work that ultimately results in in components making up Linux containers starts at Google, soon utilised by Babysitter (long running stable jobs) and Global Work Queue (batch jobs) Not much is published about them. Google starts working on Borg around 2003-2004, basing on the large prior art in scheduling but adapting it to new realities of large node counts in non-HPC tasks. A lot of the work filters into upstreamed kernel components over time, providing the basis for linux containers, with Google developing "LMCTFY" at some point as container runtime. However, they do not perform anything like Docker images. Google was not the only place that noticed such issues - we had more "cluster specific" and less generic (could be considered transitional technology) cluster managers associated with Hadoop and similar, but also several (still closed, papers published) projects from Microsoft, Alibaba, and others. Some stuff leaks, some is discussed under "how to run those jobs efficiently". 2006 comes and google upstreams some of the kernel features they developed for their system as cgroups. Linux containers project starts in 2008, IIRC clearly inspired by Solaris Zones which debut in 2005. Both still replicate VM instances, if in a "thin" manner (see also OpenVZ, Linux-VServer, BSD Jails). Amazon EC2 starts in 2006, causing huge change in how one could run their software on the internet. Curiously enough, early days EC2 is mostly stateless, this spurs "12 factor" movement further :) In few years, first custom autoscaling, then built-in AWS ASG (2009) turn up, pushing the approach of pre-baked golden VM images with your software ready to run (further pushing "12 factors", as waiting for scripts to reconfigure the service is costly when you scale up!). Sometime around that time various PaaS services, including Heroku, start building up, requiring ways to schedule customer code across machines. Google App Engine goes GA few months after start of Heroku private beta, with somewhat similar approaches (Google, being Google, is a bit more alien with the custom datastore). Availability of huge amounts of machines that aren't tightly controlled supercomputers spurs research into clustered resource management, especially since workload-specific designs like those centered around Hadoop are too specific, papers from places running internal proprietary schedulers hint at usability of generic ones. Sometime around 2009? (correct me if I'm wrong, I can't find the start date for Mesos as research project) work starts at UCB for a system that can run diverse job workloads on shared cluster, increasing resource utilization efficiency. I can't find anything reliable on whether the team knows of parallel work at google on borg, but Mesos is probably the first generic system of that class that is open sourced. Uses kernel parts of Borg if unknowingly, doesn't play with packaging. First publications 2010~2011. Twitter becomes one of the first non-research users, running on scale of hundred workers. CloudFoundry starts Warden around that time (unfortunately, Cloud Foundry is very... galapagos-experience for me, so no further details) 2013 is a big year, sees publication of Docker, then still using LXC as backend IIRC, which brings us the image metaphor for packaging. LMCTFY supposedly also goes public around that time (but I don't recall if it was indeed opensourced then, but timeline mostly fits). Oldest presentation discussing cluster management at Google happens at LISA'13 and EuroSys 2013, discussing Omega (not sure if they mention Borg by name). I'm pretty sure the name "borg" is in limited circulation by around that time, but details are still mostly secret. Definite cross-pollination of ideas between different groups at that time, even if most stuff is under NDA ;) Google also publishes "Datacenter as a Computer". Cambrian explosion of container-based schedulers follows, as people liberate themselves from one-app-per-instance paradigm. (personal aside: I suspect half of it is due to finding Docker horrible in actual deployment, as was my experience in 2014 :D). I'm probably missing details on some (OpenShift pre-3.0?) Kubernetes starts based on experiences with Borg and Omega, reusing Docker in place of LMCTFY - at least part of the reasoning is obvious, as LMCTFY (and Borg and Omega) had no means to actually provide files to run (a task that was performed by unrelated software at Google). Mesos IIRC didn't originally include anything to distribute applications either. This is (IMHO) the core innovation provided by Docker, integrating packaging into our idea of container. By 2015 containers moved seriously into public eye instead of being research project or internal implementation detail and adoption takes steam. Kubernetes 1.0 is released, and continues at insane speed (good to read the borg paper and see their pros & cons page, as it relates to this!). Initial set of features doesn't allow replicating Heroku-style PaaS, but IngressControllers debut in 1.4 (maybe 1.3?) based on OpenShift Routes, bringing HTTP routing. Enterprising admins can use it to build a PaaS, but it definitely takes assembly. Over the next few years tooling for deployment as well as various standard components one might need become more available. It becomes easier to assemble something resembling a PaaS out of kubernetes, but that's because other people write the missing bits (Ingress Controllers, service meshes, logging handlers, etc.). Alternative runtimes build and die in the frothing frenzy of development. And finally we have last few years, Mesos effectively dying, and most "warehouse-scale computer operating system" tasks ending up incorporating kubernetes, including various PaaS projects moving onto it as base substrate. IMHO that's due to kubernetes API design allowing it to move at speeds that weren't exactly possible for others, and also due to its more generic design (since it came at it from "generic workload scheduler" rather than PaaS angle) meaning that it was applicable in more areas than something designed originally to run a website, for example.