4 ms·

I love monorepos but I'm not sure if Git is the right tool beyond certain scale. Where I work doing a simple `git status` takes seconds due to the size of the r

by msoad 2y ago

I love monorepos but I'm not sure if Git is the right tool beyond certain scale. Where I work doing a simple `git status` takes seconds due to the size of the repo. There has been various attempts to solve Git performance but so far this is nothing close to what I experienced at Google. The Git team should really invest in tooling for very large repos. Our repo is around 10M files and 100M lines of code and no amount of hacks on top of Git (cache, sparse checkout etc etc) is not really solving the core problem. Meta and Google have really solved this problem internally but there is no real open source solution that works for everyone out there.

dijit 2y ago
I’m secretly hoping that google releases piper (and Mondrian); the gaming industry would go wild. Perforce is pretty brutal, and the code review tools are awful - but its still the undisputed king of mixed text and binary assets in a huge monorepo.
- habosa 2y ago
  Mondrian! That’s a name I haven’t heard in a while. Google uses Critique for code review these days. I tried to bring the best of Critique to GitHub with https://codeapprove.com https://codeapprove.com but you’re right there’s a lot that just doesn’t work on top of git.
- ralph84 2y ago
  There were rumors that at one point piper included source code licensed from Perforce. That could make open sourcing it more difficult if any of that code is still hanging around.
vlovich123 2y ago
Meta open-sourced their complete stack: https://github.com/facebook/sapling https://github.com/facebook/sapling Microsoft released Scalar (https://github.com/microsoft/scalar https://github.com/microsoft/scalar) although it's not a complete stack yet but it is already planning on releasing the backend components eventually. Have you tried Sapling? It has EdenFS baked in so it'll only materialize the files you touch and operations are fast because it has a filesystem watcher for activity so it doesn't need to do a lot of work to maintain a view of what has been invalidated.
- IshKebab 2y ago
  > Meta open-sourced their complete stack I don't think so. Last I checked there are still server components that are too tied into Facebook's infrastructure to open source. > Scalar As far as I understand it Scalar is no longer active because most of the features have been merged into mainline Git.
  - aseipp 2y ago
    FWIW, both Mononoke (the server) and EdenFS (the virtual filesystem) are now "usable, but undocumented & unsupported" in the open-source repo as of the past ~3 months, and there are published binary artifacts created on each commit if you go look through the workflows. So, if you're adventurous, you can try using them now. That said, it's probably still a year out or so from actually being usable and documented.
    - vlovich123 2y ago
      That’s what I thought from quickly skimming the repo. Thought I was a bit crazy.
brown9-2 2y ago
fsmonitor and untrackedCache doesn’t help?
optymizer 2y ago
We at Meta use Sapling, which is open source. Using EdenFS is what makes it super fast at scale. https://sapling-scm.com/ https://sapling-scm.com/ https://engineering.fb.com/2023/06/27/developer-tools/meta-developer-tools-open-source/ https://engineering.fb.com/2023/06/27/developer-tools/meta-d...

by msoad 2y ago

dijit 2y ago

I’m secretly hoping that google releases piper (and Mondrian); the gaming industry would go wild. Perforce is pretty brutal, and the code review tools are awful - but its still the undisputed king of mixed text and binary assets in a huge monorepo.

habosa 2y ago
Mondrian! That’s a name I haven’t heard in a while. Google uses Critique for code review these days. I tried to bring the best of Critique to GitHub with https://codeapprove.com https://codeapprove.com but you’re right there’s a lot that just doesn’t work on top of git.
ralph84 2y ago
There were rumors that at one point piper included source code licensed from Perforce. That could make open sourcing it more difficult if any of that code is still hanging around.

vlovich123 2y ago

Meta open-sourced their complete stack: https://github.com/facebook/sapling https://github.com/facebook/sapling Microsoft released Scalar (https://github.com/microsoft/scalar https://github.com/microsoft/scalar) although it's not a complete stack yet but it is already planning on releasing the backend components eventually. Have you tried Sapling? It has EdenFS baked in so it'll only materialize the files you touch and operations are fast because it has a filesystem watcher for activity so it doesn't need to do a lot of work to maintain a view of what has been invalidated.

IshKebab 2y ago
> Meta open-sourced their complete stack I don't think so. Last I checked there are still server components that are too tied into Facebook's infrastructure to open source. > Scalar As far as I understand it Scalar is no longer active because most of the features have been merged into mainline Git.
- aseipp 2y ago
  FWIW, both Mononoke (the server) and EdenFS (the virtual filesystem) are now "usable, but undocumented & unsupported" in the open-source repo as of the past ~3 months, and there are published binary artifacts created on each commit if you go look through the workflows. So, if you're adventurous, you can try using them now. That said, it's probably still a year out or so from actually being usable and documented.
  - vlovich123 2y ago
    That’s what I thought from quickly skimming the repo. Thought I was a bit crazy.

brown9-2 2y ago

fsmonitor and untrackedCache doesn’t help?

optymizer 2y ago

We at Meta use Sapling, which is open source. Using EdenFS is what makes it super fast at scale. https://sapling-scm.com/ https://sapling-scm.com/ https://engineering.fb.com/2023/06/27/developer-tools/meta-developer-tools-open-source/ https://engineering.fb.com/2023/06/27/developer-tools/meta-d...