4 ms·

Nice work, really interesting blog post! On a sidenote, git itself can also get painfully slow with large monorepos. Hope GitHub can push some changes there as

Nice work, really interesting blog post! On a sidenote, git itself can also get painfully slow with large monorepos. Hope GitHub can push some changes there as well. I know FB moved off git to mercurial because of performance issues.

klodolph 6y ago
My understanding is that neither Git nor Mercurial can do this well out of the box, and FB and Google both have their own extensions to Mercurial to make this possible (because even though Mercurial is often slower than Git, it’s extensible) e.g. https://facebook.github.io/watchman/ https://facebook.github.io/watchman/ - used as part of Facebook’s Mercurial solution, I think.
- vtbassmatt 6y ago
  Git also has a file system monitor interface which can use Watchman. We (GitHub) are working on a native file system monitor implementation in addition - https://github.com/gitgitgadget/git/pull/900 https://github.com/gitgitgadget/git/pull/900.
- jauer 6y ago
  And then from mercurial extensions to our own server, mononoke, which apparently has been moved under the Eden umbrella: https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden
- jayd16 6y ago
  I thought Google used some custom fork of perforce.
  - klodolph 6y ago
    From what I understand, Piper is not a fork of Perforce, but instead a completely different system with the same interface. You know, built on top of a BigTable or Spanner cluster instead of whatever Perforce uses. The Mercurial extensions are then an alternative client for Piper.
    - dlp211 6y ago
      It's my understanding that it started as a fork of perforce, Google bought a source license for perforce. Whether you'd still call a fork of perforce anymore is like asking if a Computer is the same computer after you've replaced all the components over the years.
      - arthurdenture 6y ago
        That was the case many years ago (extremely customized perforce server), but after that there was a full rewrite as a distributed system built on Bigtable etc. - https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... is an overview. It wasn't a ship-of-Theseus type situation where they incrementally replaced components.
pitaj 6y ago
You might be interested in scalar [1] developed by Microsoft for handling large repos. [1]: https://github.com/microsoft/scalar https://github.com/microsoft/scalar
- WorldMaker 6y ago
  It's also interesting to note how much of Microsoft's work for handling large repos in git has merged upstream directly into git itself. One very interesting part of that is the effort that has gone into the git commit-graph: https://git-scm.com/docs/commit-graph https://git-scm.com/docs/commit-graph. It's part of what makes scalar interesting compared to some of the projects you hear mentioned used inside the FB and Google gates: not only is scalar itself open source, but a lot of what scalar does is tune configuration flags to turn on optional git features such as the commit-graph, sparse checkout "cones", etc that are all themselves directly supported by the git client. Even if you aren't at the scale where it makes sense to use all of the tools that scalar provides, you can get some interesting baby steps by following scalar's "advice" on git configuration.
rmasters 6y ago
If you are willing to adapt to a different structure and workflow, you can filter the scope of git down dramatically with sparse checkouts (as @WorldMaker also mentioned). https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...
- chgibb 6y ago
  Sparse-checkouts are amazing. I wrote some small tools that use dependency information in Flutter packages to drive a sparse-checkout. We use it at $dayjob now.
fishywang 6y ago
Google's approach to that is to make submodules better, so you can have a "superproject" which has submodules of multiple repos that looks like a monorepo. If you look at https://android.googlesource.com/ https://android.googlesource.com/ all the projects named as "superproject" are for this purpose. Or at least that was the plan at 2016 when I left Google.

klodolph 6y ago

My understanding is that neither Git nor Mercurial can do this well out of the box, and FB and Google both have their own extensions to Mercurial to make this possible (because even though Mercurial is often slower than Git, it’s extensible) e.g. https://facebook.github.io/watchman/ https://facebook.github.io/watchman/ - used as part of Facebook’s Mercurial solution, I think.

vtbassmatt 6y ago
Git also has a file system monitor interface which can use Watchman. We (GitHub) are working on a native file system monitor implementation in addition - https://github.com/gitgitgadget/git/pull/900 https://github.com/gitgitgadget/git/pull/900.
jauer 6y ago
And then from mercurial extensions to our own server, mononoke, which apparently has been moved under the Eden umbrella: https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden
jayd16 6y ago
I thought Google used some custom fork of perforce.
- klodolph 6y ago
  From what I understand, Piper is not a fork of Perforce, but instead a completely different system with the same interface. You know, built on top of a BigTable or Spanner cluster instead of whatever Perforce uses. The Mercurial extensions are then an alternative client for Piper.
  - dlp211 6y ago
    It's my understanding that it started as a fork of perforce, Google bought a source license for perforce. Whether you'd still call a fork of perforce anymore is like asking if a Computer is the same computer after you've replaced all the components over the years.
    - arthurdenture 6y ago
      That was the case many years ago (extremely customized perforce server), but after that there was a full rewrite as a distributed system built on Bigtable etc. - https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... is an overview. It wasn't a ship-of-Theseus type situation where they incrementally replaced components.

pitaj 6y ago

You might be interested in scalar [1] developed by Microsoft for handling large repos. [1]: https://github.com/microsoft/scalar https://github.com/microsoft/scalar

WorldMaker 6y ago
It's also interesting to note how much of Microsoft's work for handling large repos in git has merged upstream directly into git itself. One very interesting part of that is the effort that has gone into the git commit-graph: https://git-scm.com/docs/commit-graph https://git-scm.com/docs/commit-graph. It's part of what makes scalar interesting compared to some of the projects you hear mentioned used inside the FB and Google gates: not only is scalar itself open source, but a lot of what scalar does is tune configuration flags to turn on optional git features such as the commit-graph, sparse checkout "cones", etc that are all themselves directly supported by the git client. Even if you aren't at the scale where it makes sense to use all of the tools that scalar provides, you can get some interesting baby steps by following scalar's "advice" on git configuration.

rmasters 6y ago

If you are willing to adapt to a different structure and workflow, you can filter the scope of git down dramatically with sparse checkouts (as @WorldMaker also mentioned). https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ https://github.blog/2020-01-17-bring-your-monorepo-down-to-s...

chgibb 6y ago
Sparse-checkouts are amazing. I wrote some small tools that use dependency information in Flutter packages to drive a sparse-checkout. We use it at $dayjob now.

fishywang 6y ago

Google's approach to that is to make submodules better, so you can have a "superproject" which has submodules of multiple repos that looks like a monorepo. If you look at https://android.googlesource.com/ https://android.googlesource.com/ all the projects named as "superproject" are for this purpose. Or at least that was the plan at 2016 when I left Google.