3 ms·

I think you're missing my point. We're on HN right, where we often discuss the difference between technical merit and market fit. There's a time and a place for

by tombh 5y ago

I think you're missing my point. We're on HN right, where we often discuss the difference between technical merit and market fit. There's a time and a place for technically correct solutions, which Google excel at, but a successful business needs more than that, it needs to capture a zeitgeist, which Google sometimes fail to do, see Google Plus for example. Of course the technical lineage of Kubernetes is Borg and parents (which I admit I've never heard of), but I put it to HN that Kubernetes' lineage is not what created the market fit. In fact I think it's disingenuous of the documentary to imply that (in fairness it does give time to Docker), hence why I'm so keen to give my version of events. Your point about how The 12 Factors apply just as well to non-containerised software is exactly the beauty of them! The 12 Factors don't come from management, they're not an attempt (at least directly) to capture profit from a market. They are somehow fundamental axioms about writing software in this current era, they're maybe even self-evident, universal truths. It turns out that there is a fundamental connection between the ability for a piece of software to be efficiently scaled and for that piece of software to be collaborated on. 2 or more developer machines share essential characteristics with 2 or more scaled containers, which share requirements with 2 or more CI instances ― reproducibility, behaviour guaranteed by tests, ease of downloading, security, configurability through ENV. You see how we felt like Heroku understood us the humble developer, it made us DevOps, all whilst helping us satisfy the bottom line of managers? Kubernetes is truly an extraordinary piece of software, but it hasn't, nor do I believe it ever will, achieve anything of The 12 Factors' lasting legacy.

p_l 5y ago
It is, in my opinion, at best converged evolution. Maybe because 12 factors are such self-evident truths - though if true, I'd at best credit them as formulation into a listicle of the fundamental behaviour of JCL batch programs on JES2/3 (don't have enough experience with other systems of that class to compare, but I suspect certain commonality). What my point is that while Heroku might be a very applicable relation to some, it's very explicitly a Platform as a Service, in a way pushing a bit back towards the model that we were escaping from (by invoking Rails name) in mid-2000 to cheap full-OS machines, from the confines "PaaSsy" LAMP. Kubernetes didn't capture a zeitgeist by introducing any of the features (though it definitely pushed containers up into enterprise world a lot). It was in no way first, or even second, despite coming from home of linux containers. It doesn't even force you to use 12 factors - I should know something about it, having done what many called stupid, as in I did "lift and shift" of over 60 "classical" applications that in no way fit 12 factors onto it. Now, trodding a bit further away from concrete facts and into more a world of personal opinion... Kubernetes captured the zeitgeist by working. There was immense desire across all kinds of business, developments, etc. for something that fit nebulous label of "private cloud". Often with local deployments involved. And to do it cheaper (or at least, easier to automate) than existing "heavy" virtualisation stacks. Not everyone could use at all, or just burn money on AWS, many VPS providers were more like classic VMware virtualization clusters in experience. Some had money to burn on Heroku and actually fit running on it (or were lucky to fit in the cheap band of usage). Thus you had OpenStack and the dwarfs: CloudStack, Eucalyptus, OpenNebula, Ganeti, with more adventurous ones building on Mesos or fully custom. All tackling the problem of, essentially, providing simpler and faster (in turnaround time) way of handling users access to resources. The most moneyed and successful (at least from pov of Ops) was OpenStack. And it was a hot mess. Upgrading an openstack cluster was a meme, and not a nice one. Meanwhile containers gain steam, for good or bad, and people start to move from moving from the quite bad "single app VM instances", to containerised approach (12 factors already hit harder due to the single-app VM approach popularised by AWS ASGs coupled with cloud-init). Into this enters kubernetes. Yes, it might seem hard to setup - but it's no OpenStack, it brings no licensing purchasing mess of VMware, it initially kills one of the more annoying aspects (though it will return) which is the overlay networking. It brings unified API, simple operating model (wish I kept the old 1.0-1.4 presentations on hand that went bottom up!). We can deploy stateless services in somewhat sensible way (and if you know how, they can be stateful too). The saner internal design and API commonality can be arguably compared to the effect of the original "heroku" command that allowed you to just deploy from a git repo - just from the point of Ops team. By next year, we move from simple ReplicationController to ReplicaSets + Deployments, as well as gaining Ingress beta and first rush of "pluggable" componentry in the form of Ingress Controllers. It's arguably the time where you can start dealing with K8s as simplistic replacement for Heroku - except it's not some closed magic operated by the Wizard of Oz, but something you can deploy on bunch of Raspberry Pi in middle of presentation at a conference after suffering a long trip (we even forgot, iirc, to bring our planned kube-master machine and had to improvise). Early versions of TPRs (priors of CRDs) make me give what were probably some of the first talks about operators (before CoreOS people coined the term) - but I used the same terminology as K8s used (Controllers). By 2017, the recommended way to deploy openstack starts to be "deploy openstack on top of k8s", and DC/OS starts to visibly lose. K8s makes certain things forced upon you, but you find that it doesn't matter and tends to make sense (Pods having one IP, etc.). Secondary/Tertiary adoption follows, bad memes start about kubernetes (like you only need it for scale, that it's cloud specific, etc. etc.) but it doesn't really matter. Soon, operators start flowing in, thanks to simplicity of the API model (this evolves later to things like Crossplane and even separating kube-apiserver into standalone program, to run without rest of k8s). K8s manifests might feel like JCL, but you don't care because of how bloody more complex the same things used to be before. Heroku might have provided easy way to deploy constrained application for a developer, but, at least in my experience, k8s became the lever with which I could move a lot of servers with little effort, whether to run large batch jobs, complex meshes of microservices, CI build jobs, or build a heroku-style PaaS. As developers learnt to follo 12 factors, the job became easier. P.S. Funnily enough, probably because accidents of history kept me on the Ops side, but IME kubernetes adoption always had push from the sysadmin side, not necessarily developer side. P.P.S. As nice as Heroku is, there's a reason why it's my main example of burning money, especially contrasting how it's billed as "cheaper than paying for an Ops engineer". In the world of global pricing, what works as cheap in Silicon Valley is... not cheap in most of the world. P.P.P.S This kinda went ranty, I guess I should maybe write a blog post on this perspective...
- tombh 5y ago
  Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look. I totally know what you mean about OpenStack! So much potential, but just way too complicated. But I'm not sure it's fair to say that Kubernetes was the first project to Just Work™. I never tried Mesos, but it's always seemed more accessible to me. But I did use Deis and Flynn though, and they were a delight in comparison. Deis of course being the home of Helm, Kubernetes' now de facto packager manager. Also don't you think that Kubernetes explicitly embraced Docker right from the start as a first-class citizen? OpenStack only ever bolted it on as an after thought. Kubernetes knew what the buzz was and made sure to let everyone know that Docker was front and centre. This is why I think it's better to think of Kubernetes as a spiritual child of Docker (and therefore everything that came before Docker) rather than a technical descendant of Borg. Although Deis and Flynn were certainly already creaking under the weight of what it truly meant to support the full gamut of cloud features, they were heading in the right direction. Google really did knock it out of the park, which I'm grateful for, but I think they did it more with the sheer weight of money and reputation than any innovative technology. Nobody really knew the devs behind Deis and Flynn, but everyone knows Google's reputation. And so we all just came to warm our hands around Kubernetes' fire, and the rest is history.
  - p_l 5y ago
    JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved. Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system). Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem. As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny. Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed. The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API). A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)

by tombh 5y ago

p_l 5y ago

It is, in my opinion, at best converged evolution. Maybe because 12 factors are such self-evident truths - though if true, I'd at best credit them as formulation into a listicle of the fundamental behaviour of JCL batch programs on JES2/3 (don't have enough experience with other systems of that class to compare, but I suspect certain commonality). What my point is that while Heroku might be a very applicable relation to some, it's very explicitly a Platform as a Service, in a way pushing a bit back towards the model that we were escaping from (by invoking Rails name) in mid-2000 to cheap full-OS machines, from the confines "PaaSsy" LAMP. Kubernetes didn't capture a zeitgeist by introducing any of the features (though it definitely pushed containers up into enterprise world a lot). It was in no way first, or even second, despite coming from home of linux containers. It doesn't even force you to use 12 factors - I should know something about it, having done what many called stupid, as in I did "lift and shift" of over 60 "classical" applications that in no way fit 12 factors onto it. Now, trodding a bit further away from concrete facts and into more a world of personal opinion... Kubernetes captured the zeitgeist by working. There was immense desire across all kinds of business, developments, etc. for something that fit nebulous label of "private cloud". Often with local deployments involved. And to do it cheaper (or at least, easier to automate) than existing "heavy" virtualisation stacks. Not everyone could use at all, or just burn money on AWS, many VPS providers were more like classic VMware virtualization clusters in experience. Some had money to burn on Heroku and actually fit running on it (or were lucky to fit in the cheap band of usage). Thus you had OpenStack and the dwarfs: CloudStack, Eucalyptus, OpenNebula, Ganeti, with more adventurous ones building on Mesos or fully custom. All tackling the problem of, essentially, providing simpler and faster (in turnaround time) way of handling users access to resources. The most moneyed and successful (at least from pov of Ops) was OpenStack. And it was a hot mess. Upgrading an openstack cluster was a meme, and not a nice one. Meanwhile containers gain steam, for good or bad, and people start to move from moving from the quite bad "single app VM instances", to containerised approach (12 factors already hit harder due to the single-app VM approach popularised by AWS ASGs coupled with cloud-init). Into this enters kubernetes. Yes, it might seem hard to setup - but it's no OpenStack, it brings no licensing purchasing mess of VMware, it initially kills one of the more annoying aspects (though it will return) which is the overlay networking. It brings unified API, simple operating model (wish I kept the old 1.0-1.4 presentations on hand that went bottom up!). We can deploy stateless services in somewhat sensible way (and if you know how, they can be stateful too). The saner internal design and API commonality can be arguably compared to the effect of the original "heroku" command that allowed you to just deploy from a git repo - just from the point of Ops team. By next year, we move from simple ReplicationController to ReplicaSets + Deployments, as well as gaining Ingress beta and first rush of "pluggable" componentry in the form of Ingress Controllers. It's arguably the time where you can start dealing with K8s as simplistic replacement for Heroku - except it's not some closed magic operated by the Wizard of Oz, but something you can deploy on bunch of Raspberry Pi in middle of presentation at a conference after suffering a long trip (we even forgot, iirc, to bring our planned kube-master machine and had to improvise). Early versions of TPRs (priors of CRDs) make me give what were probably some of the first talks about operators (before CoreOS people coined the term) - but I used the same terminology as K8s used (Controllers). By 2017, the recommended way to deploy openstack starts to be "deploy openstack on top of k8s", and DC/OS starts to visibly lose. K8s makes certain things forced upon you, but you find that it doesn't matter and tends to make sense (Pods having one IP, etc.). Secondary/Tertiary adoption follows, bad memes start about kubernetes (like you only need it for scale, that it's cloud specific, etc. etc.) but it doesn't really matter. Soon, operators start flowing in, thanks to simplicity of the API model (this evolves later to things like Crossplane and even separating kube-apiserver into standalone program, to run without rest of k8s). K8s manifests might feel like JCL, but you don't care because of how bloody more complex the same things used to be before. Heroku might have provided easy way to deploy constrained application for a developer, but, at least in my experience, k8s became the lever with which I could move a lot of servers with little effort, whether to run large batch jobs, complex meshes of microservices, CI build jobs, or build a heroku-style PaaS. As developers learnt to follo 12 factors, the job became easier. P.S. Funnily enough, probably because accidents of history kept me on the Ops side, but IME kubernetes adoption always had push from the sysadmin side, not necessarily developer side. P.P.S. As nice as Heroku is, there's a reason why it's my main example of burning money, especially contrasting how it's billed as "cheaper than paying for an Ops engineer". In the world of global pricing, what works as cheap in Silicon Valley is... not cheap in most of the world. P.P.P.S This kinda went ranty, I guess I should maybe write a blog post on this perspective...

tombh 5y ago
Very interesting about JCL being an older precedent for the 12 Factors, I've never used it, so I'll have to take a look. I totally know what you mean about OpenStack! So much potential, but just way too complicated. But I'm not sure it's fair to say that Kubernetes was the first project to Just Work™. I never tried Mesos, but it's always seemed more accessible to me. But I did use Deis and Flynn though, and they were a delight in comparison. Deis of course being the home of Helm, Kubernetes' now de facto packager manager. Also don't you think that Kubernetes explicitly embraced Docker right from the start as a first-class citizen? OpenStack only ever bolted it on as an after thought. Kubernetes knew what the buzz was and made sure to let everyone know that Docker was front and centre. This is why I think it's better to think of Kubernetes as a spiritual child of Docker (and therefore everything that came before Docker) rather than a technical descendant of Borg. Although Deis and Flynn were certainly already creaking under the weight of what it truly meant to support the full gamut of cloud features, they were heading in the right direction. Google really did knock it out of the park, which I'm grateful for, but I think they did it more with the sheer weight of money and reputation than any innovative technology. Nobody really knew the devs behind Deis and Flynn, but everyone knows Google's reputation. And so we all just came to warm our hands around Kubernetes' fire, and the rest is history.
- p_l 5y ago
  JCL and JES2/3 provided a bunch of features (of course, very primitive compared to anything newer) that helped run programs in a world before expectations of "wasting precious disk space" on configuration or state that wasn't critical, thus it tended to operate remarkably similar to running 12 factor batch jobs - for example, you would use Data Definition (origin of Unix dd command - it was a joke on JCL syntax, iirc) to specify open files (with named handles) passed to a program (consider passing that through environment variables or parameters in 12 factor app), telling it to run an otherwise stateless binary, and possibly embedding a short command program (interpreted by it) in JCL stream. A program might be preinstalled on disk, or pulled from tape, your data might be mostly on tapes, etc., and your JCL on a bunch of cards that you submit is, effectively, your declaration of environment for it and the system operators take care it's run properly - but outside of that JCL, there might be no other state involved. Kubernetes might seem to embrace Docker, but in reality it hid the Docker. In fact, kubernetes explicitly avoided using features from docker outside of certain basics - I'd even say it's part of what I liked the most, as I found Docker to be incredibly clunky once past "local development" stage. As early as 2016 we had alternative runtimes that took in docker images (and sometimes not even them) and ran them in completely different matter. It was just a question of time, when Docker will be eliminated from k8s stack, rather than if. The only real thing that k8s took from Docker and not somewhere else was the idea of container images as packaging format (containers having been widely used at google with LMCTFY and other runtimes before, and LXC predating Docker in wider world with even primitive package system). Mesos had the issue that, at least in my experience, every bit of the stack was separate. Running "just Mesos" wasn't really an option, and every framework seemed to be building with its own APIs, creating a disconnected mess (mind you, my first actual use of Mesos came after already using k8s for a while). What comes as common bits (without operators) in k8s 1.4, involved IIRC 2-3 different frameworks with disjoint APIs on DC/OS. Quite a different experience, even if you used DC/OS which packaged it altogether for you. Mind you, DC/OS arguably also totally killed community around Mesos in my PoV, but that's a question to someone deeper involved in that ecosystem. As for OpenStack vs. K8s... I can't even compare the scale difference of running "reasonably operational" k8s cluster vs. "openstack that isn't just devstack on dev's laptop". And when it comes to upgrading it really becomes unfunny. Also, at least early on, it really didn't seem that K8s would become the half ton gorilla it is now, and while people might have not known devs behind Deis and Flynn, it was in my experience heavily the evangelism of early adopters that pushed it, later bolstered by GKE becoming a thing. Despite that and google's supposed reputation (they really, really don't know how to monetize it, instead burning it - otherwise GCP would be much more of a success commercially than it is, not third in public perception of people I talk with), alternative runtimes continued on and people still believed that DC/OS might be the winner, or that a bunch of other platforms might maybe standardize some bits around docker but no clear winner might appear. Admittedly I haven't heard of Flynn, but I do recall Deis, pre-k8s Rancher, Docker Swarm, CoreOS' Fleet, the unlamented OpenStack thingy, and others being regularly discussed. The one thing that IMO made k8s truly groundbreaking was the kube-apiserver design, from first principles. Yes, a whole cluster is complex, but it builds from simple component ideas, with the simple patterns of apiserver tying everything together (said "simplicity" is grounded thoroughly in control theory and, I wonder if anyone noticed, symbolic AI designs like Blackboard Systems). This meant that evolving that system, first all internally through combined controller binary, then through TPRs and finally CRDs, was easy. Outwardly (there's some extension if you look close) the base k8s API didn't change since version 1.0, despite huge changes in what you could do with it. Something that I don't think any other alternative took care of, with exception of maybe generalized look at AWS APIs (if you squint to blur the inconsistency and try not to notice gaps hinting at private APIs and differences between what's doable from Console and API). A bunch of that came explicitly from having experience of much more haphazardly designed Borg APIs and their ossification, and it sometimes showed in discussions on github repo (I recall early requests for CPU pinning being refused by understandably vague but useful explanation of how it led to broken design in borg api and that they would recommend shelving till a good interface gets designed)