3 ms·

My understanding is that Meta/Google had to rewrite tons of stuff (distributed file systems, CVS, search tools, build tools...) and need many teams to maintain

My understanding is that Meta/Google had to rewrite tons of stuff (distributed file systems, CVS, search tools, build tools...) and need many teams to maintain all these systems, and sometime their developer experience is inferior to what you get with off-the-shelf OSS. I'm not super familiar with this topic as I don't work with systems of that scale, but I'm wondering was it worth it or necessary? what would be the alternative?

hahaxdxd123 4y ago
> sometime their developer experience is inferior to what you get with off-the-shelf OSS. The internal version control system that I use (Fig, which is apparently based off Mercurial) is so good I've basically never had to think while using it. I can't really say the same about git. CodeSearch is also, AFAIK, a lot better than what you would get OSS. I think the productivity gains across 100K engineers is definitely worth it.
Shish2k 4y ago
> sometime their developer experience is inferior to what you get with off-the-shelf OSS Working at Meta, being able to use Sapling (our Mercurial fork) is actually one of the highlights and I was pretty miserable every time I needed to go back to Git for my open-source work — thankfully it was open-sourced with a git backend a couple of weeks ago, so now I get to combine the excellent sapling CLI with the github hosting service :D
skuzye 4y ago
I have to say, I really enjoy using Google monorepo and all tools available to me. Having said that, it's not free of course. There are many engineering teams and computer power being invested to provide this environment to all googlers.
twblalock 4y ago
I've seen what happens when an org tried to adopt a Google-style monorepo without making the investments in build tooling and cultural change. It was a disaster. All of those things are needed even at orgs much smaller than Google, or you will end up with an unbuildable, unmaintainable, unreleasable mess. For orgs that can't make those investments, I think a repo per team is the best approach. Each team can treat their repo like their own little Google-style monorepo if they want to.
tylerhou 4y ago
A lot of the work is only because of the large size of the codebase. E.g. Forge and ObjFS, which is necessary because compiling the really large binaries on a normal workstation would OOM it. Or take days. https://bazel.build/basics/distributed-builds https://bazel.build/basics/distributed-builds If your codebase is "normal-sized," you don't need nearly that amount of infrastructure. There is probably some growing pain when transitioning from normal-sized to "huge," but that's part of the growing pain for any startup. You're going to have to hire people to work on internal tooling anyway; setting up a distributed build and testing service (especially now there are so many open-source and hosted implementations) is worth the effort once you're starting to scale. You're going to have to set that up regardless of a mono-repo or many separate repos. It's probably only worth hiring serious, dedicated teams that work on building like Google once your CI costs are a significant portion of operation. That probably won't happen for a while for most startups. https://github.com/bazelbuild/bazel-buildfarm https://github.com/bazelbuild/bazel-buildfarm https://cloud.google.com/build https://cloud.google.com/build https://aws.amazon.com/codebuild/ https://aws.amazon.com/codebuild/ https://azure.microsoft.com/en-us/products/devops/pipelines/ https://azure.microsoft.com/en-us/products/devops/pipelines/
- rtpg 4y ago
  I think that's a bit misleading (disclaimer: I very much like Bazel existing, though I think a better version of it could exist somewhere). Surely a lot of work is put into Bazel core to support huge workflows. But a huge amount of work is put into simply getting tools to work in hermetic environments! Especially web stack tooling is so bad about this that lots of Bazel tools are automatically patching generated scripts from npm or pip, in order to get things working properly. There is also incidental complexity when it comes to running Bazel itself, because it uses symlinks to support sandboxing by default. I have run into several programs that do "is file" checks that think that symlinks are not files. Granted, we are fortunate that lots of that work happens in the open and goes beyond Google's "just vendor it" philosophy. But Docker's "let's put it all in one big ball of mud" strategy papers over a lot of potential issues that you have to face front-on with Bazel. Once it works though... beautiful stuff.
  - 4y ago

hahaxdxd123 4y ago

> sometime their developer experience is inferior to what you get with off-the-shelf OSS. The internal version control system that I use (Fig, which is apparently based off Mercurial) is so good I've basically never had to think while using it. I can't really say the same about git. CodeSearch is also, AFAIK, a lot better than what you would get OSS. I think the productivity gains across 100K engineers is definitely worth it.

Shish2k 4y ago

> sometime their developer experience is inferior to what you get with off-the-shelf OSS Working at Meta, being able to use Sapling (our Mercurial fork) is actually one of the highlights and I was pretty miserable every time I needed to go back to Git for my open-source work — thankfully it was open-sourced with a git backend a couple of weeks ago, so now I get to combine the excellent sapling CLI with the github hosting service :D

skuzye 4y ago

I have to say, I really enjoy using Google monorepo and all tools available to me. Having said that, it's not free of course. There are many engineering teams and computer power being invested to provide this environment to all googlers.

twblalock 4y ago

I've seen what happens when an org tried to adopt a Google-style monorepo without making the investments in build tooling and cultural change. It was a disaster. All of those things are needed even at orgs much smaller than Google, or you will end up with an unbuildable, unmaintainable, unreleasable mess. For orgs that can't make those investments, I think a repo per team is the best approach. Each team can treat their repo like their own little Google-style monorepo if they want to.

tylerhou 4y ago

A lot of the work is only because of the large size of the codebase. E.g. Forge and ObjFS, which is necessary because compiling the really large binaries on a normal workstation would OOM it. Or take days. https://bazel.build/basics/distributed-builds https://bazel.build/basics/distributed-builds If your codebase is "normal-sized," you don't need nearly that amount of infrastructure. There is probably some growing pain when transitioning from normal-sized to "huge," but that's part of the growing pain for any startup. You're going to have to hire people to work on internal tooling anyway; setting up a distributed build and testing service (especially now there are so many open-source and hosted implementations) is worth the effort once you're starting to scale. You're going to have to set that up regardless of a mono-repo or many separate repos. It's probably only worth hiring serious, dedicated teams that work on building like Google once your CI costs are a significant portion of operation. That probably won't happen for a while for most startups. https://github.com/bazelbuild/bazel-buildfarm https://github.com/bazelbuild/bazel-buildfarm https://cloud.google.com/build https://cloud.google.com/build https://aws.amazon.com/codebuild/ https://aws.amazon.com/codebuild/ https://azure.microsoft.com/en-us/products/devops/pipelines/ https://azure.microsoft.com/en-us/products/devops/pipelines/

rtpg 4y ago
I think that's a bit misleading (disclaimer: I very much like Bazel existing, though I think a better version of it could exist somewhere). Surely a lot of work is put into Bazel core to support huge workflows. But a huge amount of work is put into simply getting tools to work in hermetic environments! Especially web stack tooling is so bad about this that lots of Bazel tools are automatically patching generated scripts from npm or pip, in order to get things working properly. There is also incidental complexity when it comes to running Bazel itself, because it uses symlinks to support sandboxing by default. I have run into several programs that do "is file" checks that think that symlinks are not files. Granted, we are fortunate that lots of that work happens in the open and goes beyond Google's "just vendor it" philosophy. But Docker's "let's put it all in one big ball of mud" strategy papers over a lot of potential issues that you have to face front-on with Bazel. Once it works though... beautiful stuff.
- 4y ago