4 ms·

As a Xoogler myself, I have always wondered about the logic of "we can't open source X because it uses too many libraries and is too integrated". The obvious an

As a Xoogler myself, I have always wondered about the logic of "we can't open source X because it uses too many libraries and is too integrated". The obvious answer is, OK, open source the libraries and refactor the integrations to make them more flexible. Reimplementing all of Borg from scratch seems crazy to me given the huge effort that went into it. Does Google want an open source cluster infrastructure or not? If yes, in what universe is it less effort to write a totally new one from scratch vs just progressively open sourcing things?

jsmthrowaway 8y ago
Indeed. The universe where you aren’t trying to replace Borg, and instead have other motives for conceiving of Kubernetes in the first place. That’s all I’m trying to say. Hadoop is nowhere close to the original thinking behind MapReduce, and Kubernetes is nowhere close to the original thinking behind Borg. The difference is Yahoo! wrote one, and the other leads us to speculate about why they’d want to offer such a good concept to the world, then ship something that so spectacularly misses that mark, particularly scalewise, but get away with it because it’s Google doing it. There are only a couple explanations, especially when you factor in that corporations always act in their interest. That’s not a moral dimension, that’s just how it works. What upside is there for Google having everyone do cluster orchestration the way they do? Really, I challenge you to think of one. When I was there, you couldn’t even say the word Borg externally (despite the original article not being the first security-based leak of it). Operations and engineering velocity was, and remains, a crucial competitive advantage for Google. Since I’ve left, there’s a clear shift toward teaching the industry how to do operations, from the Borg paper on through to the SRE book. Why? Why suddenly cede that competitive advantage, the ability to ship a whole production service in like, a weekend? Because you do it poorly, drive people to Google Cloud because administering Kubernetes at any scale is like self-flagellating with a rake (don’t get me started and be satisfied that I’m speaking from ulcers here), and keep ahead of the competition who are forced to respond to GKE now that a whole community is clamoring for Docker in prod. It’s the only explanation that fits. Maybe it wasn’t even intentional, but it’s what has played out. Another possible explanation is that Google wanted to end run Mesos, which was threatening to dominate until Kubernetes got a bunch of mindshare and venture capital. Mesos, with care and careful choice of schedulers, can be on the same field as Borg. Twitter is very happy with it at scales that would decimate Kubernetes, as well as many others I cannot name. Google’s VC endeavors have been dumping into Kubernetes startups, and now you don’t hear about Mesos any more. Just like Docker before it, we just can’t get enough bad operational systems as a profession, apparently. My whole career I’ve been obsessed with high scale orchestration because it is a uniquely difficult problem. I saw Borg and went Jesus, we are all behind the curve as an industry. Then the first few releases of Kubernetes came out and I killed it dead with a few dozen nodes and moderate load. It’s gotten better, yes, but Borg has literally centuries of engineering in it, and a steering committee is going to push Kubernetes to OpenStack or Java status (I’d argue it’s already there, given that people are now shipping distributions of it). And now operations as a discipline suffers for it. I can’t get a SRE gig any more without a dev team who doesn’t need Kubernetes, at all, making it a showstopper to ship. Kinda funny, by the way, that this thread has been off the front page for hours and is still active with new faces. I must have ruffled eng-misc. :)
jsnell 8y ago
What's the size of the transitive dependency graph of Borg? 10MLOC? 50MLOC? 100MLOC? I have no idea. But it's a lot of code no matter what. Open sourcing that much code is a huge undertaking, unless you're just planning to throw it over the wall with no expectation of external people working on it. On the other hand starting from scratch you get to grow the community and the codebase in lockstep.
- repolfx 8y ago
  It may be a large undertaking but yes, it's clearly still less work to release code that exists and build a community around it, than rewrite it all from scratch and also build a community around that too.
  - gpshead 8y ago
    You just filled up my BINGO card by saying "clearly" to indicate that you have no idea what you are talking about.
    - repolfx 8y ago
      I've worked at Google for many years, and built open source communities based on code I've written from scratch several times. This is not an area I'm inexperienced in.
justicezyx 8y ago
Generally Google software has a bottom up completely different approach to industry norm/standard. And the divergence started from Google’s very beginning. Open sourcing system software from its internal state requires the same amount of work as rewriting, plus the effort to morph interfaces and internals to fit external needs, plus changes to internal workloads (assuming a unified stack internal & external).
- repolfx 8y ago
  I worked on google3 for years, most likely some of the code I wrote is still there. I've also done a lot of open source coding too. I'm quite familiar with the structure of google3 and it's not as different as you claim - Borg is a bunch of C++ libraries and programs that depend on each other, nothing magic about that. So I completely disagree that open sourcing code is as hard as rewriting it from scratch. I think if you tried to argue that to anyone outside the Google bubble they'd think you were crazy. Writing code is hard work! Uploading it to github and creating some project structures around it is vastly less work. I can't help wondering if this is engineers looking for new promotion-worthy projects.
  - justicezyx 8y ago
    To rebuttal your statements, I seem need to reveal a lot of technical details. You did not mention what type of software you were open sourcing when you were in Google. But it seems our overlap in knowledge is rather small. I'll leave this open. But I want to emphasize that what I stated are reasonable reasons for open sourcing by writing from scratch.

jsmthrowaway 8y ago

Indeed. The universe where you aren’t trying to replace Borg, and instead have other motives for conceiving of Kubernetes in the first place. That’s all I’m trying to say. Hadoop is nowhere close to the original thinking behind MapReduce, and Kubernetes is nowhere close to the original thinking behind Borg. The difference is Yahoo! wrote one, and the other leads us to speculate about why they’d want to offer such a good concept to the world, then ship something that so spectacularly misses that mark, particularly scalewise, but get away with it because it’s Google doing it. There are only a couple explanations, especially when you factor in that corporations always act in their interest. That’s not a moral dimension, that’s just how it works. What upside is there for Google having everyone do cluster orchestration the way they do? Really, I challenge you to think of one. When I was there, you couldn’t even say the word Borg externally (despite the original article not being the first security-based leak of it). Operations and engineering velocity was, and remains, a crucial competitive advantage for Google. Since I’ve left, there’s a clear shift toward teaching the industry how to do operations, from the Borg paper on through to the SRE book. Why? Why suddenly cede that competitive advantage, the ability to ship a whole production service in like, a weekend? Because you do it poorly, drive people to Google Cloud because administering Kubernetes at any scale is like self-flagellating with a rake (don’t get me started and be satisfied that I’m speaking from ulcers here), and keep ahead of the competition who are forced to respond to GKE now that a whole community is clamoring for Docker in prod. It’s the only explanation that fits. Maybe it wasn’t even intentional, but it’s what has played out. Another possible explanation is that Google wanted to end run Mesos, which was threatening to dominate until Kubernetes got a bunch of mindshare and venture capital. Mesos, with care and careful choice of schedulers, can be on the same field as Borg. Twitter is very happy with it at scales that would decimate Kubernetes, as well as many others I cannot name. Google’s VC endeavors have been dumping into Kubernetes startups, and now you don’t hear about Mesos any more. Just like Docker before it, we just can’t get enough bad operational systems as a profession, apparently. My whole career I’ve been obsessed with high scale orchestration because it is a uniquely difficult problem. I saw Borg and went Jesus, we are all behind the curve as an industry. Then the first few releases of Kubernetes came out and I killed it dead with a few dozen nodes and moderate load. It’s gotten better, yes, but Borg has literally centuries of engineering in it, and a steering committee is going to push Kubernetes to OpenStack or Java status (I’d argue it’s already there, given that people are now shipping distributions of it). And now operations as a discipline suffers for it. I can’t get a SRE gig any more without a dev team who doesn’t need Kubernetes, at all, making it a showstopper to ship. Kinda funny, by the way, that this thread has been off the front page for hours and is still active with new faces. I must have ruffled eng-misc. :)

jsnell 8y ago

What's the size of the transitive dependency graph of Borg? 10MLOC? 50MLOC? 100MLOC? I have no idea. But it's a lot of code no matter what. Open sourcing that much code is a huge undertaking, unless you're just planning to throw it over the wall with no expectation of external people working on it. On the other hand starting from scratch you get to grow the community and the codebase in lockstep.

repolfx 8y ago
It may be a large undertaking but yes, it's clearly still less work to release code that exists and build a community around it, than rewrite it all from scratch and also build a community around that too.
- gpshead 8y ago
  You just filled up my BINGO card by saying "clearly" to indicate that you have no idea what you are talking about.
  - repolfx 8y ago
    I've worked at Google for many years, and built open source communities based on code I've written from scratch several times. This is not an area I'm inexperienced in.

justicezyx 8y ago

Generally Google software has a bottom up completely different approach to industry norm/standard. And the divergence started from Google’s very beginning. Open sourcing system software from its internal state requires the same amount of work as rewriting, plus the effort to morph interfaces and internals to fit external needs, plus changes to internal workloads (assuming a unified stack internal & external).

repolfx 8y ago
I worked on google3 for years, most likely some of the code I wrote is still there. I've also done a lot of open source coding too. I'm quite familiar with the structure of google3 and it's not as different as you claim - Borg is a bunch of C++ libraries and programs that depend on each other, nothing magic about that. So I completely disagree that open sourcing code is as hard as rewriting it from scratch. I think if you tried to argue that to anyone outside the Google bubble they'd think you were crazy. Writing code is hard work! Uploading it to github and creating some project structures around it is vastly less work. I can't help wondering if this is engineers looking for new promotion-worthy projects.
- justicezyx 8y ago
  To rebuttal your statements, I seem need to reveal a lot of technical details. You did not mention what type of software you were open sourcing when you were in Google. But it seems our overlap in knowledge is rather small. I'll leave this open. But I want to emphasize that what I stated are reasonable reasons for open sourcing by writing from scratch.