6 ms·

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly c

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here is not Piper per se. Unlike Microsoft, Google also has a distributed, caching, incremental build system (Blaze), and a distributed test system (Forge), and they are integrated with Piper. The vast majority of the code you depend on never actually ends up on your machine. Thanks to this, what takes hours at Microsoft takes seconds at Google. This enables pretty staggering productivity gains. You don't think twice about kicking off a build, and in most cases no more than a minute or two later you have your binaries, irrespective of the size of your transitive closure. Some projects take longer than that to build, most take less time. Tests are heavily parallelized. Dependencies are tracked (so tests can be re-run when dependencies change), there are large scale refactoring tools that let you make changes that affect the entire monorepo with confidence and without breaking anyone. Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit.

kevincox 10y ago
I think this would be better the piper+citc. While the virtual filesystem aspect is nice the perforce model is far inferior to git's. (IMO) Of course Google has tools on top of it. But it's not fair to compare a VCS and interface to it to a complete development infrastructure. Hell, with the content addressing of git it would even make it easy to build something similar.
- general_ai 10y ago
  > perforce model is far inferior to git's That's just, like, your opinion, man. There are other bits of infra that integrate with it quite nicely, and would integrate with something like Git quite poorly. One of those things is their code review system. The closest thing I could find to it outside Google is Gerrit, but it's a tremendous pain to set up and use, and it's but a pale shadow of Google's internal tool (Critique). And also, one does not preclude another: Google has a git wrapper on top of Piper, so you can spend your entire Google career not even touching Piper directly if that's what you prefer. And Piper went beyond the "Perforce model" in ways I can't disclose here.
  - piotrkaminski 10y ago
    Check out Reviewable, it's influenced by my fond memories of Critique but designed specifically for git (and GitHub).
    - nh2 10y ago
      Reviewable is amazing, thanks for making that. I've used lots of review tools and worked a bit on Google's review tool and on ReviewBoard in the past, and Reviewable is better than all of them in my opinion (or at least, better than when I last used the others).
  - codesnik 10y ago
    I wonder how do you know what parts you CAN disclose?
    - general_ai 10y ago
      Whatever is already public. You can watch a video about Piper and some other systems on YouTube, and read about other things from Google's own blogs, papers, etc.
d0vs 10y ago
> Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit. This may be naive but why not recreate it as an open source project?
- general_ai 10y ago
  Because very few people have a need to support a billion-LOC monorepo on which 30K engineers make tens of thousands of commits daily. That's where this system shines. For smaller projects, Git+Bazel (open source, non-distributed version of Blaze) works fine if you're working with C++, and other build systems work OK as well, if you're working with other languages.
- kentonv 10y ago
  Blaze has been: https://bazel.build/ https://bazel.build/ Forge and Piper are built on Google's internal tech stack and designed for Google's production infrastructure, so open sourcing them would be a very big project. I think it would be a lot more likely for them to be offered as a service -- and that might be more useful to users anyway, since you'd be able to share resources with everyone else doing builds, rather than try to get your own cluster running which might sit idle a lot of the time. Of course, there are privacy issues, etc. (Disclaimer: I'm purely speculating. I left Google over four years ago, and have no idea what the tools people are up to today.)
kentonv 10y ago
> Every single ex-Googler misses it quite a bit. I dunno. I don't miss the 1-minute incremental builds. (Maybe they've improved since I left, though.) BTW Forge is not just the test runner, but the thing that runs all build tasks, farmed out to all servers. Blaze interprets the build language and does dependency tree analysis but then hands off the tasks to Forge. Blaze has been (partially) open sourced: https://bazel.build/ https://bazel.build/
- general_ai 10y ago
  I know. :-)
jeremyepling 10y ago
I'm a Microsoft employee on the Git team. We do have a distributed, caching, incremental build system and a distributed test system. Right now, they're completely internal - like Google. They're called CloudBuild and CloudTest. They're very fast and no one thinks twice about kicking off a build.
- mooman219 10y ago
  Google employs a distributed, caching, incremental build system and a distributed test system across the majority of their code base. I worked in Windows Store and I can assure you that most people there don't use CloudBuild and CloudTest, let along know what they are. I would be confident in saying the majority of people at Microsoft are in that boat.
  - general_ai 10y ago
    +1. I know some insiders at MS and their build/test/deployment story is universally very crappy. Things barely work, held together by curse words and duct tape. Googlers like to joke internally that Google looks like a race car from the outside and like Moving Castle from Hayao Miyazaki'a cartoon from the inside, but that's not the case at all. Comparatively speaking it's a race car inside and out, it's just that the insiders don't know how shitty things are elsewhere. P.S. I heard Bing is different, but I have no visibility into it, so can't comment.
    - christophilus 10y ago
      I worked in the cloud at MS roughly 7 years ago, and Bing was very different from anything else. The MS cloud services were a total mess. I mean, as in hard to overstate how bad. In contrast, Bing was a well-oiled machine. Simple build, simple deploy, simple devops, simple test, very consistent processes across all of their teams. At the time, Azure was a joke (partly due to the fact that the initial teams were headed up by ex Office devs with no cloud experience, if I remember correctly). But Azure was cannibalizing the Bing team pretty hard. I hear that strategy worked and that Azure is in much more capable hands now.

kevincox 10y ago

I think this would be better the piper+citc. While the virtual filesystem aspect is nice the perforce model is far inferior to git's. (IMO) Of course Google has tools on top of it. But it's not fair to compare a VCS and interface to it to a complete development infrastructure. Hell, with the content addressing of git it would even make it easy to build something similar.

general_ai 10y ago
> perforce model is far inferior to git's That's just, like, your opinion, man. There are other bits of infra that integrate with it quite nicely, and would integrate with something like Git quite poorly. One of those things is their code review system. The closest thing I could find to it outside Google is Gerrit, but it's a tremendous pain to set up and use, and it's but a pale shadow of Google's internal tool (Critique). And also, one does not preclude another: Google has a git wrapper on top of Piper, so you can spend your entire Google career not even touching Piper directly if that's what you prefer. And Piper went beyond the "Perforce model" in ways I can't disclose here.
- piotrkaminski 10y ago
  Check out Reviewable, it's influenced by my fond memories of Critique but designed specifically for git (and GitHub).
  - nh2 10y ago
    Reviewable is amazing, thanks for making that. I've used lots of review tools and worked a bit on Google's review tool and on ReviewBoard in the past, and Reviewable is better than all of them in my opinion (or at least, better than when I last used the others).
- codesnik 10y ago
  I wonder how do you know what parts you CAN disclose?
  - general_ai 10y ago
    Whatever is already public. You can watch a video about Piper and some other systems on YouTube, and read about other things from Google's own blogs, papers, etc.

d0vs 10y ago

> Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit. This may be naive but why not recreate it as an open source project?

general_ai 10y ago
Because very few people have a need to support a billion-LOC monorepo on which 30K engineers make tens of thousands of commits daily. That's where this system shines. For smaller projects, Git+Bazel (open source, non-distributed version of Blaze) works fine if you're working with C++, and other build systems work OK as well, if you're working with other languages.
kentonv 10y ago
Blaze has been: https://bazel.build/ https://bazel.build/ Forge and Piper are built on Google's internal tech stack and designed for Google's production infrastructure, so open sourcing them would be a very big project. I think it would be a lot more likely for them to be offered as a service -- and that might be more useful to users anyway, since you'd be able to share resources with everyone else doing builds, rather than try to get your own cluster running which might sit idle a lot of the time. Of course, there are privacy issues, etc. (Disclaimer: I'm purely speculating. I left Google over four years ago, and have no idea what the tools people are up to today.)

kentonv 10y ago

> Every single ex-Googler misses it quite a bit. I dunno. I don't miss the 1-minute incremental builds. (Maybe they've improved since I left, though.) BTW Forge is not just the test runner, but the thing that runs all build tasks, farmed out to all servers. Blaze interprets the build language and does dependency tree analysis but then hands off the tasks to Forge. Blaze has been (partially) open sourced: https://bazel.build/ https://bazel.build/

general_ai 10y ago
I know. :-)

jeremyepling 10y ago

I'm a Microsoft employee on the Git team. We do have a distributed, caching, incremental build system and a distributed test system. Right now, they're completely internal - like Google. They're called CloudBuild and CloudTest. They're very fast and no one thinks twice about kicking off a build.

mooman219 10y ago
Google employs a distributed, caching, incremental build system and a distributed test system across the majority of their code base. I worked in Windows Store and I can assure you that most people there don't use CloudBuild and CloudTest, let along know what they are. I would be confident in saying the majority of people at Microsoft are in that boat.
- general_ai 10y ago
  +1. I know some insiders at MS and their build/test/deployment story is universally very crappy. Things barely work, held together by curse words and duct tape. Googlers like to joke internally that Google looks like a race car from the outside and like Moving Castle from Hayao Miyazaki'a cartoon from the inside, but that's not the case at all. Comparatively speaking it's a race car inside and out, it's just that the insiders don't know how shitty things are elsewhere. P.S. I heard Bing is different, but I have no visibility into it, so can't comment.
  - christophilus 10y ago
    I worked in the cloud at MS roughly 7 years ago, and Bing was very different from anything else. The MS cloud services were a total mess. I mean, as in hard to overstate how bad. In contrast, Bing was a well-oiled machine. Simple build, simple deploy, simple devops, simple test, very consistent processes across all of their teams. At the time, Azure was a joke (partly due to the fact that the initial teams were headed up by ex Office devs with no cloud experience, if I remember correctly). But Azure was cannibalizing the Bing team pretty hard. I hear that strategy worked and that Azure is in much more capable hands now.