9 ms·

From Google to the world: the Kubernetes origin story

159 points by striking 8y ago

dagenix 8y ago
Its an interesting article. Though, I kinda feel like we're missing a good chunk of the story. As dominate as AWS is now, it was probably even more dominate in the time period we're talking about due to really haven gotten a big head start. And every time that AWS rolled out some new proprietary feature, it tied users even more tightly to the AWS platform. IMO, what Google realized they needed was a technology prevent people from getting more dependent on AWS. Being open source is key to that goal - if they just offer another proprietary service, its hard to go to someone and tell them not to use an AWS proprietary service but to instead use a GCP one. However, with Kubernetes being open, it means that users can host their own cluster and make it easy for them to move between cloud providers. For customers on AWS, I feel like this provides a bit of an incentive not to use too many AWS proprietary services or otherwise they lose the advantage of mobility. For everyone else, GCP gets to offer the strongest Kubernetes implementation and draw customers in that way. Its pretty win-win for Google. The article talks about the decision to open source Kubernetes as some sort of decision for the betterment of the world. But if that were really the only reason, why not open source Spanner of BigTable? And that's not to say that Google did anything wrong - corporations are designed to make decisions that are in their own best interests. What I think is interesting, is that this is a really powerful example of how sometimes things that are generally beneficial to the world AND business interests line up together, and when that does, kinda cool things can happen.
- SkyRocknRoll 8y ago
  Well written.
- ben_jones 8y ago
  Good points, I have my own inferences too: At the time of k8s inception GCP was behind AWS in UX and features. App engine had (very arguably) failed, if only from its well deserved high potential. The allure of kubernetes tied people over while Google played catch-up. Now many of Google's services are superior to AWS (very arguably). Developers notice the better network performance, the improved GCP console UI, the GCP mobile app, etc. Now with kubernetes maturing to the point of production usability Google's sales team is starting to outperform AWS. AWS will have dominate market share for many years to come, but GCP is in the drivers seat. The generous free tier of GCP (both the $300 credits and the free-tiers of various services) are a big help as well.
- jchw 8y ago
  >why not open source Spanner [or] BigTable? Because it's genuinely difficult to open source things like that. They're way too tied to Google and have many closed-source dependencies on internal libraries and services. I think they regret missing the opportunity on some of these and that may reflect why Google has started open sourcing more 'core' things like gRPC and Bazel - it might make it easier.
  - dagenix 8y ago
    Good points. But, they also faced the same issue with Borg - presumably it was hard to open source, so, they created Kubernetes instead. There is nothing stopping Google from doing the same thing with Spanner or BigTable. Such a move would probably be well received by the community. It seems likely that people at Google would want to work on such projects - AIUI, CockrochDB was created by ex-Googlers. (And I didn't mean to focus on those technologies too specifically - with HBase and CockroachDB already existing, maybe Google wouldn't want to compete, but, I suspect they have other technologies for which creating open source versions would be well received, but they choose not to). My point isn't that Google could directly open source those technologies. Nor is it that they should create open source versions of them. I'm more just musing on how when the selfish reasons to do something line up with the good reasons to do something, you tend to get cool results. At a previous job, we had this process that everyone was supposed to do via a script. Doing the process via the script meant we got the logging that we wanted and such. Everyone was supposed to do it this way. However, there were no technical controls preventing anyone from doing that process by hand. We just told people not to. So, one day, a colleague asks me how to make sure that everyone does the process the right way. And my answer is that everyone already does - the script is way easier to run that to do it by hand. The good thing to do (run it the way you were told to), lined up with the selfish reason (it was way easier to do that way). Making sure those two things were in alignment was effectively as powerful as making it impossible to do the wrong way. (since then, the process has been significantly changed). I feel like its a similar case here. The good thing to do was to create a cool technology and make it free for everyone. The selfish thing to do, was to create a technology that made business sense. The result, is Kubernetes was created as open source (at least, thats my theory). If that wasn't the good thing to do, maybe the person writing the article doesn't talk to the VP on the shuttle about it and it never happens. If it wasn't the selfish thing to do (business case), maybe the VP wouldn't approve it. But, getting those two things aligned, results in positive reinforcement and results.
- barrkel 8y ago
  Yup. Kubernetes is a clear platform play to commoditize AWS, their biggest competitor in the cloud platform space. The strategy wouldn't have worked without open sourcing it, because it relied on people building k8s clusters in AWS, even if AWS didnt support it directly. Leaving out the major business rationale for k8s makes the article seem childish.
- swaroop 8y ago
  Agreed! This was also explained nicely in https://stratechery.com/2016/how-google-cloud-platform-is-challenging-aws/ https://stratechery.com/2016/how-google-cloud-platform-is-ch...
- dekhn 8y ago
  leveldb with a server wrapped around it is substantially like bigtable.
  - Xorlev 8y ago
    If you really want Bigtable, go use Apache HBase. It's very mature these days and performed really well at $dayjob-1. There's a lot that goes into turning LevelDB into a real database. :)
- thesausageking 8y ago
  > The article talks about the decision to open source Kubernetes as some sort of decision for the betterment of the world. I've never liked that Google feels the need to couch everything in making the world better. Their open source strategy has mainly been strategic business moves to stop competitive threats. Android was about stopping Apple from dominating mobile OSs. Kubernetes was about stopping AWS from dominating cloud. TensorFlow was about making sure Nvidia didn't dominate deep learning.
  - stingraycharles 8y ago
    This is precisely on point and is a proven strategy for Google. Chrome is another example. Almost all their big open source projects are to mitigate existing threats, and it’s working out very well for them. It’s annoying to see them trying to frame it as a “public service”, but guess that’s how to do proper PR.
  - xapata 8y ago
    Google purchased Android before the iPhone was created. I agree that open source is good business, but some of the causes you listed aren't the whole story. Also, they learned a lesson from Hadoop. If they'd open-sourced Google MapReduce, the cloud industry might look very different.
  - tangentspace 8y ago
    I don't get any odd vibe like that from this article. They are pretty upfront about the business motivations in this post: > with the launch of our Infrastructure-as-a-Service platform Google Compute Engine, we noticed an interesting problem: customers were paying for a lot of CPUs, but their utilization rates were extremely low because they were running VMs. We knew we had an internal solution for this. Later, > most importantly, we were able to work with lots of great engineers, many of whom really understood the needs of businesses who would benefit from deploying containers That does not read like a decision to pursue this for the betterment of the world. It sounds like a decision to pursue this for the betterment of their customers. That's capitalism, not benevolence, and they're pretty transparent about it.
rhymenoceros 8y ago
Title needs a (2016).
jonny_eh 8y ago
Wait, so Kubernetes is Google's take on Docker, invented independently?
- tybit 8y ago
  No it’s Googles take on container/compute task scheduling, deciding when and on which machines a task (e.g a docker container) should run. Docker Swarm is Dockers offering in the same category as Kubernetes. Edit: Though Google could be credited with introducing containers to Linux, as they added the main missing piece, cgroups, to the Linux kernel.
  - throw2016 8y ago
    Cgroups are not container specific and are used to limit resources to processes ie cpu, memory. Namespaces are what makes Linux containers possible.
telltruth 8y ago
I am bit tired by these VP-speak articles. So what really was the discussion with Eric Brewer? What arguments made him suddenly agreed to open source? Why was Urs rejected this before? Why are you not allowed to write about this? This whole article is like “I talked to my VP and convinced him to open source internal tool. We at Google are the best and everything we have build is the best and here’s the link for free trial.” And how the heck did you got “number of years” worth of coding number?
- amarkov 8y ago
  Most business ideas aren't rejected for any kind of compact reason. The decisionmaker typically has other things that are higher priority, or they just kinda avoid giving a green light until it's interpreted as a rejection.
- mlazos 8y ago
  Yeah I felt like barfing when he said “Good ideas usually win out at Google, and we were convinced this was a good idea.” It’s like dude you’re using an anecdote about running into a VP on a shuttle ride and saying this? Google is such a magic non-political place.
keypusher 8y ago
A turning point was a fateful shuttle ride where I found myself sitting next to Eric Brewer, VP of Cloud, [...] Soon after, we got the green light from Urs. This seems like a red flag for management at Google, if the best way to pitch ideas up the chain is hoping you can ambush someone important on the company shuttle.
- traskjd 8y ago
  You say red flag, others might say 'how the world really works'. There's a lot more luck and talking to the right person than we'd like to admit in this world.
  - mpweiher 8y ago
    To me "this is a red flag" and "how the world really works" are not incompatible statement.
pi-squared 8y ago
> ... And, on top of that, you want to open source it? (speculative continuation) > But if we do that and make people believe that containers - the massive, heavy, broken abstraction - are the future, and provide complicated infrastructure that will magically fix the problems of containers, and then also provide this complicated infrastructure managed - this will be our way to beat AWS. It will be difficult and clumsy to set it up on your own - you need many machines to set up the cluster so that it's "Google-scalable" and "fault-tolerant" so for most people and companies it will be way too much hassle and too expensive to manage their own cluster purely on VM or physical instances. We "just" provide you with the best managed infra, because c'mon - it will be open source and even given up control on paper - but everyone will associate it with us, we will make sure to have podcasts, and blogs, and marketing that talk about containers, the future and how G created this project. Best people will help us build it "out in the open" and then when we hire them, it will be easy to teach them this Borg monstrosity that we have here. So you know win-win-win - devs think they solved their Docker-is-shit problem with the magic of Kubernetes (yeah, I got a name for it already) so G is now savior; it's a win for GCP; it's a win for hiring. > [Urs]: Now we talkin...
- tnolet 8y ago
  That made me chuckle. I’m afraid it might even be half true. Someone should write a good blog post on how Kubernetes is a tech marketing success story. Same could be said for Docker of course.
iamgopal 8y ago
I am soon to venture in to SAAS business. Should I be interested in kubernetes ? Is old way of doing things still relevant?
- brianwawok 8y ago
  I would look hard at k8s or serverless. I went with k8s for my SaaS business and it is a huge competitive edge.
  - Aduket 8y ago
    what do you mean by "competitive edge", do you regret choosing it? and why?
    - brianwawok 8y ago
      Competitive edge. Like it helps me beat my competition. My competitors deploy once a week. I deploy 30x a day. I get CI/CD for near free, which is amazing. All else equal between us, this will let me beat them to market. The opposite of regrets. I suggest k8s or serverless for almost anyone...
- tnolet 8y ago
  This completely depends on your business. Are you running many cloud based workloads in a Dockerized manner? Are you having trouble orchestrating, monitoring and scaling those workloads? Then use Kubernetes. In most if not all other cases, Heroku will probably get you through the first couple of years.
- borplk 8y ago
  If you are indie avoid it until you have 100 users and some revenue. The last thing you want is to be fiddling with k8s when you have 3 whole users and $0 revenue. Unless your niche strongly requires fancy architecture start with a $5 VPS and take it up from there. You will never look back and think "shit if only I had used k8s on day one it wouldn't have failed". More often it's closer to "shit if I had focused on non-tech stuff more maybe it would have gone somewhere ... instead I spent 2 months fiddling with YAML files".
eloff 8y ago
The strategy appears to be working, I notice GitLab announced a migration from Azure to GCP and the reason given is to have better Kubernetes support. https://about.gitlab.com/2018/06/25/moving-to-gcp/ https://about.gitlab.com/2018/06/25/moving-to-gcp/
- ATsch 8y ago
  I'm sure the fact that Google ventures is afaik gitlab's biggest investor has a lot to do with it too.
  - williamchia 8y ago
    GV operates independently of Google (part of why the name changed officially form “Google Ventures” to GV) GitLab parters with and supports multiple clouds - e.g. GitLab announced official support for Amazon EKS, and IBM Cloud runs GitLab Core for version control.
    - dekhn 8y ago
      GV isn't truly independent. Best way to describe it is that multiple companies in Alphabet have access to Google's tech stack and resource manager. In the early days, I advised against GV trying to get its funded companies to GCP because it wasn't mature enough and I wanted the startups to just be successful, not be political statements about Google's cloud. That changed over the past few years; now I would easily recommend that startups funded by GV use Google Cloud if they wanted to.

159 points by striking 8y ago

Its an interesting article. Though, I kinda feel like we're missing a good chunk of the story. As dominate as AWS is now, it was probably even more dominate in the time period we're talking about due to really haven gotten a big head start. And every time that AWS rolled out some new proprietary feature, it tied users even more tightly to the AWS platform. IMO, what Google realized they needed was a technology prevent people from getting more dependent on AWS. Being open source is key to that goal - if they just offer another proprietary service, its hard to go to someone and tell them not to use an AWS proprietary service but to instead use a GCP one. However, with Kubernetes being open, it means that users can host their own cluster and make it easy for them to move between cloud providers. For customers on AWS, I feel like this provides a bit of an incentive not to use too many AWS proprietary services or otherwise they lose the advantage of mobility. For everyone else, GCP gets to offer the strongest Kubernetes implementation and draw customers in that way. Its pretty win-win for Google. The article talks about the decision to open source Kubernetes as some sort of decision for the betterment of the world. But if that were really the only reason, why not open source Spanner of BigTable? And that's not to say that Google did anything wrong - corporations are designed to make decisions that are in their own best interests. What I think is interesting, is that this is a really powerful example of how sometimes things that are generally beneficial to the world AND business interests line up together, and when that does, kinda cool things can happen.

SkyRocknRoll 8y ago
Well written.
ben_jones 8y ago
Good points, I have my own inferences too: At the time of k8s inception GCP was behind AWS in UX and features. App engine had (very arguably) failed, if only from its well deserved high potential. The allure of kubernetes tied people over while Google played catch-up. Now many of Google's services are superior to AWS (very arguably). Developers notice the better network performance, the improved GCP console UI, the GCP mobile app, etc. Now with kubernetes maturing to the point of production usability Google's sales team is starting to outperform AWS. AWS will have dominate market share for many years to come, but GCP is in the drivers seat. The generous free tier of GCP (both the $300 credits and the free-tiers of various services) are a big help as well.
jchw 8y ago
>why not open source Spanner [or] BigTable? Because it's genuinely difficult to open source things like that. They're way too tied to Google and have many closed-source dependencies on internal libraries and services. I think they regret missing the opportunity on some of these and that may reflect why Google has started open sourcing more 'core' things like gRPC and Bazel - it might make it easier.
- dagenix 8y ago
  Good points. But, they also faced the same issue with Borg - presumably it was hard to open source, so, they created Kubernetes instead. There is nothing stopping Google from doing the same thing with Spanner or BigTable. Such a move would probably be well received by the community. It seems likely that people at Google would want to work on such projects - AIUI, CockrochDB was created by ex-Googlers. (And I didn't mean to focus on those technologies too specifically - with HBase and CockroachDB already existing, maybe Google wouldn't want to compete, but, I suspect they have other technologies for which creating open source versions would be well received, but they choose not to). My point isn't that Google could directly open source those technologies. Nor is it that they should create open source versions of them. I'm more just musing on how when the selfish reasons to do something line up with the good reasons to do something, you tend to get cool results. At a previous job, we had this process that everyone was supposed to do via a script. Doing the process via the script meant we got the logging that we wanted and such. Everyone was supposed to do it this way. However, there were no technical controls preventing anyone from doing that process by hand. We just told people not to. So, one day, a colleague asks me how to make sure that everyone does the process the right way. And my answer is that everyone already does - the script is way easier to run that to do it by hand. The good thing to do (run it the way you were told to), lined up with the selfish reason (it was way easier to do that way). Making sure those two things were in alignment was effectively as powerful as making it impossible to do the wrong way. (since then, the process has been significantly changed). I feel like its a similar case here. The good thing to do was to create a cool technology and make it free for everyone. The selfish thing to do, was to create a technology that made business sense. The result, is Kubernetes was created as open source (at least, thats my theory). If that wasn't the good thing to do, maybe the person writing the article doesn't talk to the VP on the shuttle about it and it never happens. If it wasn't the selfish thing to do (business case), maybe the VP wouldn't approve it. But, getting those two things aligned, results in positive reinforcement and results.
barrkel 8y ago
Yup. Kubernetes is a clear platform play to commoditize AWS, their biggest competitor in the cloud platform space. The strategy wouldn't have worked without open sourcing it, because it relied on people building k8s clusters in AWS, even if AWS didnt support it directly. Leaving out the major business rationale for k8s makes the article seem childish.
swaroop 8y ago
Agreed! This was also explained nicely in https://stratechery.com/2016/how-google-cloud-platform-is-challenging-aws/ https://stratechery.com/2016/how-google-cloud-platform-is-ch...
dekhn 8y ago
leveldb with a server wrapped around it is substantially like bigtable.
- Xorlev 8y ago
  If you really want Bigtable, go use Apache HBase. It's very mature these days and performed really well at $dayjob-1. There's a lot that goes into turning LevelDB into a real database. :)
thesausageking 8y ago
> The article talks about the decision to open source Kubernetes as some sort of decision for the betterment of the world. I've never liked that Google feels the need to couch everything in making the world better. Their open source strategy has mainly been strategic business moves to stop competitive threats. Android was about stopping Apple from dominating mobile OSs. Kubernetes was about stopping AWS from dominating cloud. TensorFlow was about making sure Nvidia didn't dominate deep learning.
- stingraycharles 8y ago
  This is precisely on point and is a proven strategy for Google. Chrome is another example. Almost all their big open source projects are to mitigate existing threats, and it’s working out very well for them. It’s annoying to see them trying to frame it as a “public service”, but guess that’s how to do proper PR.
- xapata 8y ago
  Google purchased Android before the iPhone was created. I agree that open source is good business, but some of the causes you listed aren't the whole story. Also, they learned a lesson from Hadoop. If they'd open-sourced Google MapReduce, the cloud industry might look very different.
- tangentspace 8y ago
  I don't get any odd vibe like that from this article. They are pretty upfront about the business motivations in this post: > with the launch of our Infrastructure-as-a-Service platform Google Compute Engine, we noticed an interesting problem: customers were paying for a lot of CPUs, but their utilization rates were extremely low because they were running VMs. We knew we had an internal solution for this. Later, > most importantly, we were able to work with lots of great engineers, many of whom really understood the needs of businesses who would benefit from deploying containers That does not read like a decision to pursue this for the betterment of the world. It sounds like a decision to pursue this for the betterment of their customers. That's capitalism, not benevolence, and they're pretty transparent about it.

rhymenoceros 8y ago

Title needs a (2016).

jonny_eh 8y ago

Wait, so Kubernetes is Google's take on Docker, invented independently?

tybit 8y ago
No it’s Googles take on container/compute task scheduling, deciding when and on which machines a task (e.g a docker container) should run. Docker Swarm is Dockers offering in the same category as Kubernetes. Edit: Though Google could be credited with introducing containers to Linux, as they added the main missing piece, cgroups, to the Linux kernel.
- throw2016 8y ago
  Cgroups are not container specific and are used to limit resources to processes ie cpu, memory. Namespaces are what makes Linux containers possible.

telltruth 8y ago

I am bit tired by these VP-speak articles. So what really was the discussion with Eric Brewer? What arguments made him suddenly agreed to open source? Why was Urs rejected this before? Why are you not allowed to write about this? This whole article is like “I talked to my VP and convinced him to open source internal tool. We at Google are the best and everything we have build is the best and here’s the link for free trial.” And how the heck did you got “number of years” worth of coding number?

amarkov 8y ago
Most business ideas aren't rejected for any kind of compact reason. The decisionmaker typically has other things that are higher priority, or they just kinda avoid giving a green light until it's interpreted as a rejection.
mlazos 8y ago
Yeah I felt like barfing when he said “Good ideas usually win out at Google, and we were convinced this was a good idea.” It’s like dude you’re using an anecdote about running into a VP on a shuttle ride and saying this? Google is such a magic non-political place.

keypusher 8y ago

A turning point was a fateful shuttle ride where I found myself sitting next to Eric Brewer, VP of Cloud, [...] Soon after, we got the green light from Urs. This seems like a red flag for management at Google, if the best way to pitch ideas up the chain is hoping you can ambush someone important on the company shuttle.

traskjd 8y ago
You say red flag, others might say 'how the world really works'. There's a lot more luck and talking to the right person than we'd like to admit in this world.
- mpweiher 8y ago
  To me "this is a red flag" and "how the world really works" are not incompatible statement.

pi-squared 8y ago

> ... And, on top of that, you want to open source it? (speculative continuation) > But if we do that and make people believe that containers - the massive, heavy, broken abstraction - are the future, and provide complicated infrastructure that will magically fix the problems of containers, and then also provide this complicated infrastructure managed - this will be our way to beat AWS. It will be difficult and clumsy to set it up on your own - you need many machines to set up the cluster so that it's "Google-scalable" and "fault-tolerant" so for most people and companies it will be way too much hassle and too expensive to manage their own cluster purely on VM or physical instances. We "just" provide you with the best managed infra, because c'mon - it will be open source and even given up control on paper - but everyone will associate it with us, we will make sure to have podcasts, and blogs, and marketing that talk about containers, the future and how G created this project. Best people will help us build it "out in the open" and then when we hire them, it will be easy to teach them this Borg monstrosity that we have here. So you know win-win-win - devs think they solved their Docker-is-shit problem with the magic of Kubernetes (yeah, I got a name for it already) so G is now savior; it's a win for GCP; it's a win for hiring. > [Urs]: Now we talkin...

tnolet 8y ago
That made me chuckle. I’m afraid it might even be half true. Someone should write a good blog post on how Kubernetes is a tech marketing success story. Same could be said for Docker of course.

iamgopal 8y ago

I am soon to venture in to SAAS business. Should I be interested in kubernetes ? Is old way of doing things still relevant?

brianwawok 8y ago
I would look hard at k8s or serverless. I went with k8s for my SaaS business and it is a huge competitive edge.
- Aduket 8y ago
  what do you mean by "competitive edge", do you regret choosing it? and why?
  - brianwawok 8y ago
    Competitive edge. Like it helps me beat my competition. My competitors deploy once a week. I deploy 30x a day. I get CI/CD for near free, which is amazing. All else equal between us, this will let me beat them to market. The opposite of regrets. I suggest k8s or serverless for almost anyone...
tnolet 8y ago
This completely depends on your business. Are you running many cloud based workloads in a Dockerized manner? Are you having trouble orchestrating, monitoring and scaling those workloads? Then use Kubernetes. In most if not all other cases, Heroku will probably get you through the first couple of years.
borplk 8y ago
If you are indie avoid it until you have 100 users and some revenue. The last thing you want is to be fiddling with k8s when you have 3 whole users and $0 revenue. Unless your niche strongly requires fancy architecture start with a $5 VPS and take it up from there. You will never look back and think "shit if only I had used k8s on day one it wouldn't have failed". More often it's closer to "shit if I had focused on non-tech stuff more maybe it would have gone somewhere ... instead I spent 2 months fiddling with YAML files".

eloff 8y ago

The strategy appears to be working, I notice GitLab announced a migration from Azure to GCP and the reason given is to have better Kubernetes support. https://about.gitlab.com/2018/06/25/moving-to-gcp/ https://about.gitlab.com/2018/06/25/moving-to-gcp/

ATsch 8y ago
I'm sure the fact that Google ventures is afaik gitlab's biggest investor has a lot to do with it too.
- williamchia 8y ago
  GV operates independently of Google (part of why the name changed officially form “Google Ventures” to GV) GitLab parters with and supports multiple clouds - e.g. GitLab announced official support for Amazon EKS, and IBM Cloud runs GitLab Core for version control.
  - dekhn 8y ago
    GV isn't truly independent. Best way to describe it is that multiple companies in Alphabet have access to Google's tech stack and resource manager. In the early days, I advised against GV trying to get its funded companies to GCP because it wasn't mature enough and I wanted the startups to just be successful, not be political statements about Google's cloud. That changed over the past few years; now I would easily recommend that startups funded by GV use Google Cloud if they wanted to.