4 ms·

An interesting article, though I was surprised by the assertion that Google regrets Hadoop and Mesos being developed and controlled by groups outside of Google.

An interesting article, though I was surprised by the assertion that Google regrets Hadoop and Mesos being developed and controlled by groups outside of Google. At least for the last decade, I got the impression that Google deliberately didn't open source their code because they saw it as a major competitive advantage. Perhaps that is changing now that data is the new competitive advantage?

wstrange 11y ago
In the case of Kubernetes, I think Google's motivation is really simple: AWS has a huge lead but it is also a proprietary platform. If they can commoditize the platform from underneath Amazon, they believe they can build out better/faster/cheaper cloud infrastructure.
jamesblonde 11y ago
Hadoop is moving the way Linux did - fast and with a large number of companies pushing it together. As big as Google is, it wouldn't be able to dominate Linux, just as it can't now dominate Hadoop. My interpretation of the author is that it would have been better for Google to have something they can control the development of, rather than have somebody outside the tent pushing it in ways that don't suit them. Look at IBM and Spark. They see a long term threat/opportunity and moved quickly to negate it.
- TheIronYuppie 11y ago
  Disclaimer: I work at Google on Kubernetes. To be clear, this is exactly why we donated 100% of Kubernetes to the Cloud Native Computing Foundation. We believe very strongly that the reason Linux was successful was that no single company controlled it. Though we have opinions about what makes container management successful, we are but a minority of the more than 650+ contributors to the platform.
jsnell 11y ago
There are multiple reasons not to open source code. One is the idea of the proprietary code as a competitive advantage. But even in the last decade there were definitely people inside Google who were arguing that the internal infrastructure, even if the best in the world at that time, was actually becoming a liability. Some obvious arguments are: 1 - You don't get to share the burden of the maintenance with other users of the code. 2 - It's guaranteed that after every acquisition the code-base of the bought company will be hard to integrate; it's been written against the publicly available infrastructure, which is of course wildly incompatible with the internal one. 3 - New hires will often already know how the open source tools work, while they'll have to learn all of the proprietary equivalents from scratch. 4 - Engineers will generally prefer to work on projects where the knowledge is transferable elsewhere. Staffing is easier if you can say with a straight face that people will not just be working with a big proprietary ball of mud, but at least partially with publicly available technology. The other reason I can think of is that open sourcing parts of existing code bases is a lot harder than it looks [0]. It gets harder if the code is in a monorepo [1]. It gets harder if there's a strong culture of library reuse. The further you get away from the center of the dependency graph, the harder it gets. Could you open source Borg without open sourcing Chubby? GFS? Babysitter? (My recollection, which might be faulty, is that even long after Babysitter was otherwise dead, it was still used for bootstrapping Borg clusters). And since the goal has to be to create a viable project rather than just a code dump, there has to be a way for outside contributors to make changes. That needs to be possible for both the project itself, and all of its dependencies. Either you need realtime synchronization between the internal and public repos, or you have to make the public one the repository of record, and pretty much give up on the concept of the monorepo. Which would have been a really hard sell at least at last decade's Google. It's a lot easier to do this for green-field projects, with plans for releasing as open source being taken into account right from the start. [0] I wrote more on that subject a year ago; https://www.snellman.net/blog/archive/2015-03-19-cant-even-throw-code-across-the-wall/ https://www.snellman.net/blog/archive/2015-03-19-cant-even-t... [1] http://danluu.com/monorepo/ http://danluu.com/monorepo/
- mentat 11y ago
  I've been joking that Kubernetes and gRPC are really to seed the startup ecosystem to make companies easier to acquire, interesting to read that here.
  - mikecb 11y ago
    Even better: huge investment in their cloud and desire to get more people to switch. Response to best-of-class technology has always resulted in cries of lock-in. Opensourcing these things completely removes that risk. Kubernetes: our container services are best; run your containers here, but if you don't want to anymore, run them anywhere using exactly the same api. gRPC: our pub/sub is the best; run your rpc here, but if you don't want to anymore, run them anywhere using the same api. tensorflow: our managed learning api is the best; run your learning here, but if you don't want to anymore, run it anywhere with the same api. etc etc with dataflow and apache beam. These are best in class services, and except for bigquery (which has been a big driver of some of the big-name moves recently), they're using this model to take away the risk of transition. It's pretty smart.

wstrange 11y ago

In the case of Kubernetes, I think Google's motivation is really simple: AWS has a huge lead but it is also a proprietary platform. If they can commoditize the platform from underneath Amazon, they believe they can build out better/faster/cheaper cloud infrastructure.

jamesblonde 11y ago

Hadoop is moving the way Linux did - fast and with a large number of companies pushing it together. As big as Google is, it wouldn't be able to dominate Linux, just as it can't now dominate Hadoop. My interpretation of the author is that it would have been better for Google to have something they can control the development of, rather than have somebody outside the tent pushing it in ways that don't suit them. Look at IBM and Spark. They see a long term threat/opportunity and moved quickly to negate it.

TheIronYuppie 11y ago
Disclaimer: I work at Google on Kubernetes. To be clear, this is exactly why we donated 100% of Kubernetes to the Cloud Native Computing Foundation. We believe very strongly that the reason Linux was successful was that no single company controlled it. Though we have opinions about what makes container management successful, we are but a minority of the more than 650+ contributors to the platform.

jsnell 11y ago

There are multiple reasons not to open source code. One is the idea of the proprietary code as a competitive advantage. But even in the last decade there were definitely people inside Google who were arguing that the internal infrastructure, even if the best in the world at that time, was actually becoming a liability. Some obvious arguments are: 1 - You don't get to share the burden of the maintenance with other users of the code. 2 - It's guaranteed that after every acquisition the code-base of the bought company will be hard to integrate; it's been written against the publicly available infrastructure, which is of course wildly incompatible with the internal one. 3 - New hires will often already know how the open source tools work, while they'll have to learn all of the proprietary equivalents from scratch. 4 - Engineers will generally prefer to work on projects where the knowledge is transferable elsewhere. Staffing is easier if you can say with a straight face that people will not just be working with a big proprietary ball of mud, but at least partially with publicly available technology. The other reason I can think of is that open sourcing parts of existing code bases is a lot harder than it looks [0]. It gets harder if the code is in a monorepo [1]. It gets harder if there's a strong culture of library reuse. The further you get away from the center of the dependency graph, the harder it gets. Could you open source Borg without open sourcing Chubby? GFS? Babysitter? (My recollection, which might be faulty, is that even long after Babysitter was otherwise dead, it was still used for bootstrapping Borg clusters). And since the goal has to be to create a viable project rather than just a code dump, there has to be a way for outside contributors to make changes. That needs to be possible for both the project itself, and all of its dependencies. Either you need realtime synchronization between the internal and public repos, or you have to make the public one the repository of record, and pretty much give up on the concept of the monorepo. Which would have been a really hard sell at least at last decade's Google. It's a lot easier to do this for green-field projects, with plans for releasing as open source being taken into account right from the start. [0] I wrote more on that subject a year ago; https://www.snellman.net/blog/archive/2015-03-19-cant-even-throw-code-across-the-wall/ https://www.snellman.net/blog/archive/2015-03-19-cant-even-t... [1] http://danluu.com/monorepo/ http://danluu.com/monorepo/

mentat 11y ago
I've been joking that Kubernetes and gRPC are really to seed the startup ecosystem to make companies easier to acquire, interesting to read that here.
- mikecb 11y ago
  Even better: huge investment in their cloud and desire to get more people to switch. Response to best-of-class technology has always resulted in cries of lock-in. Opensourcing these things completely removes that risk. Kubernetes: our container services are best; run your containers here, but if you don't want to anymore, run them anywhere using exactly the same api. gRPC: our pub/sub is the best; run your rpc here, but if you don't want to anymore, run them anywhere using the same api. tensorflow: our managed learning api is the best; run your learning here, but if you don't want to anymore, run it anywhere with the same api. etc etc with dataflow and apache beam. These are best in class services, and except for bigquery (which has been a big driver of some of the big-name moves recently), they're using this model to take away the risk of transition. It's pretty smart.