6 ms·

It's important to distinguish between the different flavors of containerization - Docker, and container orchestration - K8S. Docker is intriguing as a nice lig

It's important to distinguish between the different flavors of containerization - Docker, and container orchestration - K8S. Docker is intriguing as a nice lightweight way of separating your processes - and is (mostly?) interesting to developers. K8S changed the entire trajectory of application management at scale. We have about 4300 containers in production where I work (at this moment - it can change from hour to hour), and I don't believe "isolated environments that need different versions of libraries" was ever a driver, or even strong incentive for using containers. Rather - the ability to rapidly add resources, resize resources, re-allocate storage performance dynamically, and use whichever cloud provider made sense (we've worked with AWS, Google Cloud, and now Azure) - and be able to migrate your environments between them in a period of a week - was what made containerization attractive. Being able to type: kubectl scale deploy worker-background --replicas=300 And have 300 worker pods come online in < 60 seconds really changes the power of an individual. Being able to have that done all dynamically (up and down) without any human intervention - that's a game changer in some of the ETL heavy workloads. If you've ever worked at a large company - think of the effort required to get 300 new 6 CPU/32 GB VMs allocated to a project you are working on. Now consider a company where pretty much all of the engineering / ops employees can do that from the bash prompt. I'm just looking forward to the day where we can scale the performance of our storage from 500 IOPS/sec to 8K IOPS/sec dynamically as required (still requires a manual step with Azure - albeit a pretty straightforward one in their portal).

jiggawatts 5y ago
This is a little bit hilarious to me, because I've been through this exact same "reinventing the wheel" cycle about 3 times now. The cycle goes like this: "A mainframe takes too long to buy, it's too expensive!" "Here, have an LPAR, you can spin one up in seconds!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get an LPAR!" "Getting an LPAR approved takes too much paperwork!" "Here, have an Intel server instead! It's not a 'proper' machine, but you can have it in a week!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new server provisioned properly!" "Getting a server provisioning approved takes too much paperwork!" "Here, have a VMware virtual server instead! It's not a 'proper' machine, but you can have it in a day!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new VM provisioned properly!" "Getting the VM provisioning approved takes too much paperwork!" "Here, have a Cloud server instead! It's not a 'proper' machine, but you can have it in minutes!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new cloud IaaS VM provisioned properly!" "Getting cloud IaaS VM provisioning approved takes too much paperwork!" "Here, have a Kubernetes Namespace instead! It's not a 'proper' machine, but you can have it in minutes!" <-- you are here.
- ghshephard 5y ago
  I totally dig what you are saying - and I've been through all of these cycles (well, our IBM minicomputer was dedicated to a single application, so we didn't have engineers spinning up LPARS) + a gig at Loudcloud/Opsware (Pre-Virtualization, so spinning up new environments was nowhere near fast enough to be considered on demand). And I've never worked at a company that let me spin up cloud computers myself (though I've obviously done so myself with Digital Ocean, et al.) I ran an IT organization that had VMware so I've been on that side - but, once again, employees could never spin up their own stuff, and requesting a new one was never a day, usually took a week or so. And I've worked on customer sites where it was a month+ to get VMs provisioned. So, despite never having worked at companies as enlightened about empowering engineers to do stuff like you have - your entire analysis has a pretty solid grain of truth, and with $2mm+ in cloud costs at the current gig - we may be entering the "Wooaa there" stage of the cycle, but, I don't ever recall having a toolset that was globally available that let people scale up pods/deploy namespaces the way we do here. We have the ability for engineers to clone an entire customer environment, databases and all, do their work on it, issue a PR, and shut down the clone in a period of 24 hours - and this happens a hundred+ times/month. An engineer interested in working in the latest release is a single click (plus filling in a couple fields around names/labels/release) + 3 minutes 30 seconds away from having a complete build of our current app deployed on 12 containers. I don't think any of the earlier cycles had that flexibility. Also - when I look at the costs - honestly, the engineering environments are essentially round-off error compared to the full scale multi-terabyte, 500+ container customer environments - so empowering them to do all these things has essentially zero real cost impact. So - while I think there is some fundamental insight into what you are saying - and I genuinely appreciate it - I think the flexibility / velocity of k8s container management / deployment (particularly around the dynamic resizing of CPU/Memory + replica count) was as readily available as it is now - though, I'm guessing an SRE qualified on Vsphere could jump in right now and tell me everything on K8S was always available in virtualization if you had the right tooling... Does cause one to think what the next cycle will entail though...
  - jiggawatts 5y ago
    > Does cause one to think what the next cycle will entail though... I've just spent a couple of years doing various small projects in AWS and Azure. They're... okay. Good, but not great. Proprietary. Closed-source. Etc, etc... All of that makes me suspect that eventually the cloud will be commoditized and end up being based on a common standard. Kubernetes may or may not become that standard. It certainly feels like the early days of POSIX and Linux starting to slowly take over the proprietary Solaris, HP-UX, and SCO Unix shops. Something to note is the Kubernetes is already a cloud-within-a-cloud. It has reinvented half of the commonly used Azure/AWS components: Resource groups, storage management, monitoring, naming, tagging, DNS, IP allocation, etc... Why should I define my resources in Azure, and then again in Kubernetes? You can see where this is leading: new cloud providers will appear, and they will be Kubernetes only. Instead of proprietary template languages, simply upload your helm chart instead. No complex virtual networking, just use IPv6 and let Kubernetes take care of the rest.

jiggawatts 5y ago

This is a little bit hilarious to me, because I've been through this exact same "reinventing the wheel" cycle about 3 times now. The cycle goes like this: "A mainframe takes too long to buy, it's too expensive!" "Here, have an LPAR, you can spin one up in seconds!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get an LPAR!" "Getting an LPAR approved takes too much paperwork!" "Here, have an Intel server instead! It's not a 'proper' machine, but you can have it in a week!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new server provisioned properly!" "Getting a server provisioning approved takes too much paperwork!" "Here, have a VMware virtual server instead! It's not a 'proper' machine, but you can have it in a day!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new VM provisioned properly!" "Getting the VM provisioning approved takes too much paperwork!" "Here, have a Cloud server instead! It's not a 'proper' machine, but you can have it in minutes!" "Wooaa there! You can't just do whatever you want, fill in this paperwork to get a new cloud IaaS VM provisioned properly!" "Getting cloud IaaS VM provisioning approved takes too much paperwork!" "Here, have a Kubernetes Namespace instead! It's not a 'proper' machine, but you can have it in minutes!" <-- you are here.

ghshephard 5y ago
I totally dig what you are saying - and I've been through all of these cycles (well, our IBM minicomputer was dedicated to a single application, so we didn't have engineers spinning up LPARS) + a gig at Loudcloud/Opsware (Pre-Virtualization, so spinning up new environments was nowhere near fast enough to be considered on demand). And I've never worked at a company that let me spin up cloud computers myself (though I've obviously done so myself with Digital Ocean, et al.) I ran an IT organization that had VMware so I've been on that side - but, once again, employees could never spin up their own stuff, and requesting a new one was never a day, usually took a week or so. And I've worked on customer sites where it was a month+ to get VMs provisioned. So, despite never having worked at companies as enlightened about empowering engineers to do stuff like you have - your entire analysis has a pretty solid grain of truth, and with $2mm+ in cloud costs at the current gig - we may be entering the "Wooaa there" stage of the cycle, but, I don't ever recall having a toolset that was globally available that let people scale up pods/deploy namespaces the way we do here. We have the ability for engineers to clone an entire customer environment, databases and all, do their work on it, issue a PR, and shut down the clone in a period of 24 hours - and this happens a hundred+ times/month. An engineer interested in working in the latest release is a single click (plus filling in a couple fields around names/labels/release) + 3 minutes 30 seconds away from having a complete build of our current app deployed on 12 containers. I don't think any of the earlier cycles had that flexibility. Also - when I look at the costs - honestly, the engineering environments are essentially round-off error compared to the full scale multi-terabyte, 500+ container customer environments - so empowering them to do all these things has essentially zero real cost impact. So - while I think there is some fundamental insight into what you are saying - and I genuinely appreciate it - I think the flexibility / velocity of k8s container management / deployment (particularly around the dynamic resizing of CPU/Memory + replica count) was as readily available as it is now - though, I'm guessing an SRE qualified on Vsphere could jump in right now and tell me everything on K8S was always available in virtualization if you had the right tooling... Does cause one to think what the next cycle will entail though...
- jiggawatts 5y ago
  > Does cause one to think what the next cycle will entail though... I've just spent a couple of years doing various small projects in AWS and Azure. They're... okay. Good, but not great. Proprietary. Closed-source. Etc, etc... All of that makes me suspect that eventually the cloud will be commoditized and end up being based on a common standard. Kubernetes may or may not become that standard. It certainly feels like the early days of POSIX and Linux starting to slowly take over the proprietary Solaris, HP-UX, and SCO Unix shops. Something to note is the Kubernetes is already a cloud-within-a-cloud. It has reinvented half of the commonly used Azure/AWS components: Resource groups, storage management, monitoring, naming, tagging, DNS, IP allocation, etc... Why should I define my resources in Azure, and then again in Kubernetes? You can see where this is leading: new cloud providers will appear, and they will be Kubernetes only. Instead of proprietary template languages, simply upload your helm chart instead. No complex virtual networking, just use IPv6 and let Kubernetes take care of the rest.