6 ms·

This reads like a giant ad for GKE. It emphasizes several times to just use GKE for pretty lame reasons (Google has good SREs and Google started the project).

This reads like a giant ad for GKE. It emphasizes several times to just use GKE for pretty lame reasons (Google has good SREs and Google started the project). The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code. I don't have an issue with GKE, but this article uses little evidence to recommend it when it seems the conclusion should have been "maintaining a k8s cluster requires a full time sysadmin. If your company has a culture of pretending sysadmins are pointless, then you should pay another company offering k8s sysadmin-as-a-service hosted on their hardware."

jrochkind1 9y ago
I dunno, if it's an ad for GKE, then "you typically have only about nine months until the latest version goes out of support" is indeed pretty persuasive marketing to me to let someone else deal with it.
- bigiain 9y ago
  This is a showstopper for me. All our best clients are on a 12 or 24 month upgrade cycles. I need to be confident I can deliver them the latest version of their project knowing there's only regular security updates required over then next 1-2 years. While I'd love to be doing continuous deployment and multiple production feature deliveries a day - that's not how the business I work at makes their money. I can live with Ubuntu's 5 year LTS policy, I can't work with a "you might have to do significant rework after just 9 months" platform.
  - jrochkind1 9y ago
    Especially cause then you've got to jump 3 significant versions, to get another 9 months of reprieve.
  - Twirrim 9y ago
    After watching the deployment team trying to keep up with Docker and their ridiculous pace of breaking changes, and then eventual decision to move entirely away from Docker for containers; I really understand their decision not to want to touch k8s with a 10 ft pole. If the choice is between "building features based around business needs so other teams can make more awesome stuff" and "throwing lots of dev time at just chasing docker/k8s/whatever", why on earth would they want to choose the latter?
- bonesss 9y ago
  I think it's important to judge the support model in context of what kubernetes is... As an API based cloud-management platform Kubernetes has a robust, stable, core topped with multiple layers of abstractions that build on, and use, one another. There are relatively few dead-ends in the projects history, and those have all tend to be superseded by a much nicer, much smarter, much easier abstractions. An operating system going out of support after 9 months is a total show stopper, naturally. But Kubernetes runs on top of those 'stable' layers. It's stability is in its primitives and the physical API contracts. The primitives being introduces today don't magically impact my production cluster, and because kubernetes is built like a russian nesting doll of APIs there is very little incentive for the project to make any changes, much less hasty changes, much less poorly thought out changes, to the core APIs. Unlike many other dependencies the kubernetes changes impact deployment and management, but will rarely impact application-level concerns. If it worked in old kubernetes, it's generally gonna work in new kubernetes. I feel absolutely no support-pressure to upgrade my on-prem installations when my deployments are cross-compatible with the updated versions on my cloud providers. I feel a lot of developer-giddiness-pressure to upgrade my on-prem installations because me and my devs want the cool new things they're baking into the platform... From my experience with k8s the issues are all related to running a linux cluster that uses docker and iptables -- a pretty unavoidable pain to run docker containers on a linux cluster, IMO. Support in this context broadly means a stable API, and a stable core, and Kubernetes has had those for a while now.
aberoham 9y ago
Author here. I do have a bias toward GKE but that is after trying to use everything else -- DIY, kubespray, OpenShift, Juju, Tectonic, Azure, Fargate, kops, Rancher v2 preview, kubeadm, etc. In my experience nothing (yet) has been as nice and clean as Google Cloud.
- iooi 9y ago
  Have you tried EKS?
  - aberoham 9y ago
    Have not, was not invited to the preview. Friends behind bootkube hinted that the EKS team seamed to be heading in directions that were odd relative to upstream. Happy to try it and be pleasantly surprised!
  - emmelaich 9y ago
    EKS = Amazon Kubernetes. (I had to look it up)
- lugg 9y ago
  Curious why swarm isn't in that list. How many services / nodes are you managing? I found the docker swarm learning curve to be very sharp but plateaued super quick, maybe a month of reading docs and bug reports then I was up to speed and don't really run into day to day issues. Once your prod is setup and you've worked out all the hard decisions around stack and how to do things, getting Devs up to speed is really easy because all they need to know is how to read the docker compose reference. Even less of any issue learning everything now too because they redesigned their documentation (wording etc as well) and it's much clearer, gotchas are explained etc.
  - sandGorgon 9y ago
    Agreed. I absolutely love Swarm and think they have done a great job of an integrated solution. However, I'm keen to see if Swarm becomes a later on top of Kubernetes instead of a separate piece if infrastructure.
rpbo 9y ago
Full disclosure: I do a lot of work with cloud providers and GCloud is one of my partners. I think the article covers a very real issue with the pace of k8s. GKE is particularly good at staying updated. There are other providers that are good like Stackpoint Cloud. If I were going to make an ad for GKE I'd point out that they probably have the most robust internal fiber network of any cloud. AWS tries to push packets onto the public internet as fast as possible. The load balancer you get with GKE is seriously overpowered and under priced (which makes me love it). Maybe EKS will be amazing (but we don't have evidence yet). Azure is getting pretty good but they're a bit late to the party. IBM isn't bad but they've struggled the most with staying up-to-date.
- bonesss 9y ago
  Haven't more recent versions of k8s (say 1.9+), dropped some self-upgrading capabilities to smooth the pain of keeping klusters kurrent? I'm a few versions back on my clusters, but my understanding was that some decent work was being done to smooth the particular pain point of keeping pace with the project :)
crb 9y ago
> The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. I happen to be in the building so I sat in on an engineering review for a proposed new GKE feature today. Tim was there, as was Brian Grant, and pretty much everyone else you've heard of (in the room or on the VC). There were also a good number of PMs, SREs, and the leads/engineers working on the GKE machinery. Thus I have to respectfully disagree.
- rpbo 9y ago
  Can confirm, this is my experience as well.
- hueving 9y ago
  Engineering review is not the same thing as reviewing and writing code. I've seen countless projects with sound designs riddled with implementation bugs due to people not fully understanding the intent of certain systems. Sitting in a room during a high level engineering review falls squarely under "limited overlap".
  - crb 9y ago
    Well, you can trust me that it doesn't, or you can send me your CV for a referral. Plus, there's this: https://www.quora.com/What-is-Googles-internal-code-review-policy-process https://www.quora.com/What-is-Googles-internal-code-review-p.... But you'll just complain that the OWNERS aren't the right people, and I'll just comment that they are.
dward 9y ago
> The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code. This statement is absolutely false. There is no upstream team separate from Google Cloud team. There is only the GKE team.
tyler_larson 9y ago
The GKE reference is a lot more relevant than you think, it's just a bit deeper below the surface. Google's relationship with k8s and GKE and in particular their motivation for creating an open-source orchestration service is well documented and frequently retold, but ironically appears to not be well-understood. With any sufficiently complex compute solution, you're going to need both a platform and some sort of management. Typically vendors sell you their management service (for which you often pay a recurring fee) by locking you into a proprietary platform. What's more, the management service can be dismal but you'll pay anyway because your app is built on their platform. It happens everywhere, the industry is full of examples, and anyone could see that it was going to hit Cloud sooner rather than later. And the result would be ... bad ... for everyone (customers and providers alike) but the dominant platform. So Google took an exceptionally gutsy gamble. Google excels at platform management. They invented SRE. They posited that in a level playing field, all else being equal, Google would have the best platform management service, and people would pay for it based on its merits alone. By making k8s open source, they leveled the playing field. It was by far the best tool for the job, and everybody could have it at no cost; neither money nor burden. So there was no business model or justification for building a lock-in sub-par alternative. The platform became a commodity; nobody could own it. Which is where we are now, and what this article depicts. You still need a management strategy, k8s doesn't change that. But now everyone's management service has to compete on its own merits. Nobody gets to play the lock-in game. Google's management service is GKE. It's good, but not because Google started Kubernetes. Rather, Google started Kubernetes because Google has the expertise and experience to make GKE good.
- bonesss 9y ago
  I see it, somewhat cynically, as an advanced competitive strategy. At some point something will become "the standard". Up to that point Google will have to be making huge investment in their own tooling to operate at their scale... If "the standard" is someone elses, they're playing second fiddle in the big picture. If "the standard" is different than their own tooling they're paying a premium in dollars and a premium in talent-time to train new hires. Giving away their tooling and supporting it to become a 'best of breed' solution outsources their training costs and 'onboarding' time to the greater industry. Facebook, MS, and VMWare are paying people to get good at Google tech and selling Googles tech in the Entprise. Their open source strategy also ensures, as you said, a 'level playing field'. A level playing field where they are guaranteed to not get locked out, and where where their size, strengths, and deep competencies of product and domain give them a massive competitive advantage. It's smart on a lot of levels.

jrochkind1 9y ago

I dunno, if it's an ad for GKE, then "you typically have only about nine months until the latest version goes out of support" is indeed pretty persuasive marketing to me to let someone else deal with it.

bigiain 9y ago
This is a showstopper for me. All our best clients are on a 12 or 24 month upgrade cycles. I need to be confident I can deliver them the latest version of their project knowing there's only regular security updates required over then next 1-2 years. While I'd love to be doing continuous deployment and multiple production feature deliveries a day - that's not how the business I work at makes their money. I can live with Ubuntu's 5 year LTS policy, I can't work with a "you might have to do significant rework after just 9 months" platform.
- jrochkind1 9y ago
  Especially cause then you've got to jump 3 significant versions, to get another 9 months of reprieve.
- Twirrim 9y ago
  After watching the deployment team trying to keep up with Docker and their ridiculous pace of breaking changes, and then eventual decision to move entirely away from Docker for containers; I really understand their decision not to want to touch k8s with a 10 ft pole. If the choice is between "building features based around business needs so other teams can make more awesome stuff" and "throwing lots of dev time at just chasing docker/k8s/whatever", why on earth would they want to choose the latter?
bonesss 9y ago
I think it's important to judge the support model in context of what kubernetes is... As an API based cloud-management platform Kubernetes has a robust, stable, core topped with multiple layers of abstractions that build on, and use, one another. There are relatively few dead-ends in the projects history, and those have all tend to be superseded by a much nicer, much smarter, much easier abstractions. An operating system going out of support after 9 months is a total show stopper, naturally. But Kubernetes runs on top of those 'stable' layers. It's stability is in its primitives and the physical API contracts. The primitives being introduces today don't magically impact my production cluster, and because kubernetes is built like a russian nesting doll of APIs there is very little incentive for the project to make any changes, much less hasty changes, much less poorly thought out changes, to the core APIs. Unlike many other dependencies the kubernetes changes impact deployment and management, but will rarely impact application-level concerns. If it worked in old kubernetes, it's generally gonna work in new kubernetes. I feel absolutely no support-pressure to upgrade my on-prem installations when my deployments are cross-compatible with the updated versions on my cloud providers. I feel a lot of developer-giddiness-pressure to upgrade my on-prem installations because me and my devs want the cool new things they're baking into the platform... From my experience with k8s the issues are all related to running a linux cluster that uses docker and iptables -- a pretty unavoidable pain to run docker containers on a linux cluster, IMO. Support in this context broadly means a stable API, and a stable core, and Kubernetes has had those for a while now.

aberoham 9y ago

Author here. I do have a bias toward GKE but that is after trying to use everything else -- DIY, kubespray, OpenShift, Juju, Tectonic, Azure, Fargate, kops, Rancher v2 preview, kubeadm, etc. In my experience nothing (yet) has been as nice and clean as Google Cloud.

iooi 9y ago
Have you tried EKS?
- aberoham 9y ago
  Have not, was not invited to the preview. Friends behind bootkube hinted that the EKS team seamed to be heading in directions that were odd relative to upstream. Happy to try it and be pleasantly surprised!
- emmelaich 9y ago
  EKS = Amazon Kubernetes. (I had to look it up)
lugg 9y ago
Curious why swarm isn't in that list. How many services / nodes are you managing? I found the docker swarm learning curve to be very sharp but plateaued super quick, maybe a month of reading docs and bug reports then I was up to speed and don't really run into day to day issues. Once your prod is setup and you've worked out all the hard decisions around stack and how to do things, getting Devs up to speed is really easy because all they need to know is how to read the docker compose reference. Even less of any issue learning everything now too because they redesigned their documentation (wording etc as well) and it's much clearer, gotchas are explained etc.
- sandGorgon 9y ago
  Agreed. I absolutely love Swarm and think they have done a great job of an integrated solution. However, I'm keen to see if Swarm becomes a later on top of Kubernetes instead of a separate piece if infrastructure.

rpbo 9y ago

Full disclosure: I do a lot of work with cloud providers and GCloud is one of my partners. I think the article covers a very real issue with the pace of k8s. GKE is particularly good at staying updated. There are other providers that are good like Stackpoint Cloud. If I were going to make an ad for GKE I'd point out that they probably have the most robust internal fiber network of any cloud. AWS tries to push packets onto the public internet as fast as possible. The load balancer you get with GKE is seriously overpowered and under priced (which makes me love it). Maybe EKS will be amazing (but we don't have evidence yet). Azure is getting pretty good but they're a bit late to the party. IBM isn't bad but they've struggled the most with staying up-to-date.

bonesss 9y ago
Haven't more recent versions of k8s (say 1.9+), dropped some self-upgrading capabilities to smooth the pain of keeping klusters kurrent? I'm a few versions back on my clusters, but my understanding was that some decent work was being done to smooth the particular pain point of keeping pace with the project :)

crb 9y ago

> The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. I happen to be in the building so I sat in on an engineering review for a proposed new GKE feature today. Tim was there, as was Brian Grant, and pretty much everyone else you've heard of (in the room or on the VC). There were also a good number of PMs, SREs, and the leads/engineers working on the GKE machinery. Thus I have to respectfully disagree.

rpbo 9y ago
Can confirm, this is my experience as well.
hueving 9y ago
Engineering review is not the same thing as reviewing and writing code. I've seen countless projects with sound designs riddled with implementation bugs due to people not fully understanding the intent of certain systems. Sitting in a room during a high level engineering review falls squarely under "limited overlap".
- crb 9y ago
  Well, you can trust me that it doesn't, or you can send me your CV for a referral. Plus, there's this: https://www.quora.com/What-is-Googles-internal-code-review-policy-process https://www.quora.com/What-is-Googles-internal-code-review-p.... But you'll just complain that the OWNERS aren't the right people, and I'll just comment that they are.

dward 9y ago

> The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code. This statement is absolutely false. There is no upstream team separate from Google Cloud team. There is only the GKE team.

tyler_larson 9y ago

The GKE reference is a lot more relevant than you think, it's just a bit deeper below the surface. Google's relationship with k8s and GKE and in particular their motivation for creating an open-source orchestration service is well documented and frequently retold, but ironically appears to not be well-understood. With any sufficiently complex compute solution, you're going to need both a platform and some sort of management. Typically vendors sell you their management service (for which you often pay a recurring fee) by locking you into a proprietary platform. What's more, the management service can be dismal but you'll pay anyway because your app is built on their platform. It happens everywhere, the industry is full of examples, and anyone could see that it was going to hit Cloud sooner rather than later. And the result would be ... bad ... for everyone (customers and providers alike) but the dominant platform. So Google took an exceptionally gutsy gamble. Google excels at platform management. They invented SRE. They posited that in a level playing field, all else being equal, Google would have the best platform management service, and people would pay for it based on its merits alone. By making k8s open source, they leveled the playing field. It was by far the best tool for the job, and everybody could have it at no cost; neither money nor burden. So there was no business model or justification for building a lock-in sub-par alternative. The platform became a commodity; nobody could own it. Which is where we are now, and what this article depicts. You still need a management strategy, k8s doesn't change that. But now everyone's management service has to compete on its own merits. Nobody gets to play the lock-in game. Google's management service is GKE. It's good, but not because Google started Kubernetes. Rather, Google started Kubernetes because Google has the expertise and experience to make GKE good.

bonesss 9y ago
I see it, somewhat cynically, as an advanced competitive strategy. At some point something will become "the standard". Up to that point Google will have to be making huge investment in their own tooling to operate at their scale... If "the standard" is someone elses, they're playing second fiddle in the big picture. If "the standard" is different than their own tooling they're paying a premium in dollars and a premium in talent-time to train new hires. Giving away their tooling and supporting it to become a 'best of breed' solution outsources their training costs and 'onboarding' time to the greater industry. Facebook, MS, and VMWare are paying people to get good at Google tech and selling Googles tech in the Entprise. Their open source strategy also ensures, as you said, a 'level playing field'. A level playing field where they are guaranteed to not get locked out, and where where their size, strengths, and deep competencies of product and domain give them a massive competitive advantage. It's smart on a lot of levels.