6 ms·

How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively? You would probably need a bit of tooling.

How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively? You would probably need a bit of tooling. But the gain is that git commands in the submodules are speedy, and there is only O(logN) commit multiplication to commit the updated commit SHAs up the chain. Think Merkle tree, not single head commit SHA. Eventually, you may get a monstrosity like Android Repo [2] though. And an Android checkout and build is pushing 1TB these days. But there, perhaps, the submodule idea wins again. Replace most of the submodules with prebuilt variants, and have full source + building only for the module of interest. [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules [2] https://source.android.com/docs/setup/download#repo https://source.android.com/docs/setup/download#repo

tazjin 3y ago
Most large monorepos simply are not on git. Google has Piper, Yandex has arc, Facebook has eden (which is actually semi-open-source, btw!), some companies use Perforce and so on.
- avidiax 3y ago
  I think it's orthogonal. "Monorepo" is a culture around having a single branch with a single lineage, and not developing anything in any isolation greater than a single developer's workstation. I agree that Git is not very adequate for large monorepos, but I'd say that most open source projects are on Git, and most of them are trivial monorepos.
  - hgsgm 3y ago
    No. Branching is orthogonal to modules/monorepo.
- 0xcoffee 3y ago
  Windows is coming out with their own 'Dev Drive': https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/ I'm very curious how it performs compared to EdenFS: https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O...
  - tazjin 3y ago
    Maybe I'm misreading those Dev Drive docs, but those don't seem related in any way? Dev Drive seems to be a special type of disk volume with higher reliability or something for dev-related workloads. EdenFS is "Facebook's CITC", i.e. a virtual filesystem view into a remote version-control system.
- MikusR 3y ago
  Microsoft uses git.
  - tazjin 3y ago
    Not off-the-shelf git though, they have their own file system virtualisation stuff on top. Some of that used to be open-source (Windows only, I think?).
    - WorldMaker 3y ago
      VFS for Git is still Open Source: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit Microsoft's blog posts have indicated a move to use something as close to off-the-shelf git as possible, though. They say they've stopped using VFS much and are instead more often relying on sparse checkouts. They've upstreamed a lot of patches into git itself, and maintain their own git fork but the fork distance is generally shrinking as those patches upstream.
maccard 3y ago
The problem with submodules is they're not "vanilla" git, and have some subtle, unexpected behaviours. See this thread[0] for some examples. Submodules, like LFS, are a great idea that suck in practice because they're bolted on to git to avoid compromising the purity of the base project. [0] https://news.ycombinator.com/item?id=31792303 https://news.ycombinator.com/item?id=31792303
- deleted 3y ago
  [deleted]
ants_everywhere 3y ago
The monorepo is essentially a single file system. Things like moving a file from one git submodule to another is more cumbersome than just `mv foo dir/bar`. That means your directory structure is in practice tightly coupled to the tree of git projects. Also, since any of the git sub-repos can be branched, the chaos of merging development branches seems like it gets even more complicated in a submodule architecture. It may be possible to put a user interface that abstracts away the submodule architecture and forces everything to live on HEAD. But at that point it might be easier to just provide a git-like UI to a centralized VCS.
jcarrano 3y ago
The monorepo is where you end up when you have failed to enforce encapsulation and your "modules" do not have stable APIs (or are actually modular). Then, with sub-modules each change will often involve multiple commits to different modules, plus commits to update references, so O(N) commit multiplication.
- hgsgm 3y ago
  Monorepo works whether your APIs are modular or not, and also allows changes to the modular structure.
solarkraft 3y ago
> And an Android checkout and build is pushing 1TB these days I remember it "only" being somewhere around 200Gb.
ramesh31 3y ago
>How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively? Submodules are essentially broken with no way to fix them. It was a good idea that never took off.
dmoy 3y ago
> How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively One fundamental way it differs is atomic commits. You can't change something in repo A and subsubrepo XYZ in a single pull. A monorepo allows you to do things like atomic commits to arbitrary pairs of files in the repo, which among other things opens up the possibility of enforcing single-version of libraries, which in turn removes a whole class of diamond dependency issues. There's other benefits, but imo it's probably not worth it for most companies because of the staggering number of things it breaks in the developer toolspace once it gets large enough. Eventually you need teams of people that do nothing but make tooling to support monorepo scaling, because everything off the shelf explodes (what do you do when even perforce can't handle your repo?) For example, at Google we have a team of people who do nothing but, effectively, recreate the cross referencing and jump-to-def everyone else gets for "free" from IntelliJ / VS intellisence, etc. (We do other stuff too, but that's a fair paraphrase). And on top of that the team really only exists because Steve Yegge is a Force to be Reckoned With, otherwise we might still be flailing around without jump to def, idk.
- jeffbee 3y ago
  The ability of Google's internal code search to jump between declaration, definition, override, and call site is miles ahead of what Intellisense can do.
- HdS84 3y ago
  One Major Point for monorepos is the ability to eschew packages. In most languages creating, publishing and consuming a package is a lot of work, while in a monorepo you just add a reference to the code and are ready to go (except in react native...gaah that was pure horror). That's especially valuable if you need to refactor something and need to adjust it's dependencies. Doing that via packages is slow and painful. Via project references it's mich easier and has a tight feature loop, change+build+fix instead of of change+build+publish+consume+fix
  - dmoy 3y ago
    Yup, agree completely. That's a natural extension of the same thing that enables atomic commits - suddenly just having direct library dependencies instead of packages isn't that big of a problem if you push everything into the monorepo. > That's especially valuable if you need to refactor something and need to adjust it's dependencies. And yes exactly, being able to change a library and all of its callers at the same time is pretty handy.

tazjin 3y ago

Most large monorepos simply are not on git. Google has Piper, Yandex has arc, Facebook has eden (which is actually semi-open-source, btw!), some companies use Perforce and so on.

avidiax 3y ago
I think it's orthogonal. "Monorepo" is a culture around having a single branch with a single lineage, and not developing anything in any isolation greater than a single developer's workstation. I agree that Git is not very adequate for large monorepos, but I'd say that most open source projects are on Git, and most of them are trivial monorepos.
- hgsgm 3y ago
  No. Branching is orthogonal to modules/monorepo.
0xcoffee 3y ago
Windows is coming out with their own 'Dev Drive': https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/ I'm very curious how it performs compared to EdenFS: https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O...
- tazjin 3y ago
  Maybe I'm misreading those Dev Drive docs, but those don't seem related in any way? Dev Drive seems to be a special type of disk volume with higher reliability or something for dev-related workloads. EdenFS is "Facebook's CITC", i.e. a virtual filesystem view into a remote version-control system.
MikusR 3y ago
Microsoft uses git.
- tazjin 3y ago
  Not off-the-shelf git though, they have their own file system virtualisation stuff on top. Some of that used to be open-source (Windows only, I think?).
  - WorldMaker 3y ago
    VFS for Git is still Open Source: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit Microsoft's blog posts have indicated a move to use something as close to off-the-shelf git as possible, though. They say they've stopped using VFS much and are instead more often relying on sparse checkouts. They've upstreamed a lot of patches into git itself, and maintain their own git fork but the fork distance is generally shrinking as those patches upstream.

maccard 3y ago

The problem with submodules is they're not "vanilla" git, and have some subtle, unexpected behaviours. See this thread[0] for some examples. Submodules, like LFS, are a great idea that suck in practice because they're bolted on to git to avoid compromising the purity of the base project. [0] https://news.ycombinator.com/item?id=31792303 https://news.ycombinator.com/item?id=31792303

deleted 3y ago
[deleted]

ants_everywhere 3y ago

The monorepo is essentially a single file system. Things like moving a file from one git submodule to another is more cumbersome than just `mv foo dir/bar`. That means your directory structure is in practice tightly coupled to the tree of git projects. Also, since any of the git sub-repos can be branched, the chaos of merging development branches seems like it gets even more complicated in a submodule architecture. It may be possible to put a user interface that abstracts away the submodule architecture and forces everything to live on HEAD. But at that point it might be easier to just provide a git-like UI to a centralized VCS.

jcarrano 3y ago

The monorepo is where you end up when you have failed to enforce encapsulation and your "modules" do not have stable APIs (or are actually modular). Then, with sub-modules each change will often involve multiple commits to different modules, plus commits to update references, so O(N) commit multiplication.

hgsgm 3y ago
Monorepo works whether your APIs are modular or not, and also allows changes to the modular structure.

solarkraft 3y ago

> And an Android checkout and build is pushing 1TB these days I remember it "only" being somewhere around 200Gb.

ramesh31 3y ago

>How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively? Submodules are essentially broken with no way to fix them. It was a good idea that never took off.

dmoy 3y ago

> How does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively One fundamental way it differs is atomic commits. You can't change something in repo A and subsubrepo XYZ in a single pull. A monorepo allows you to do things like atomic commits to arbitrary pairs of files in the repo, which among other things opens up the possibility of enforcing single-version of libraries, which in turn removes a whole class of diamond dependency issues. There's other benefits, but imo it's probably not worth it for most companies because of the staggering number of things it breaks in the developer toolspace once it gets large enough. Eventually you need teams of people that do nothing but make tooling to support monorepo scaling, because everything off the shelf explodes (what do you do when even perforce can't handle your repo?) For example, at Google we have a team of people who do nothing but, effectively, recreate the cross referencing and jump-to-def everyone else gets for "free" from IntelliJ / VS intellisence, etc. (We do other stuff too, but that's a fair paraphrase). And on top of that the team really only exists because Steve Yegge is a Force to be Reckoned With, otherwise we might still be flailing around without jump to def, idk.

jeffbee 3y ago
The ability of Google's internal code search to jump between declaration, definition, override, and call site is miles ahead of what Intellisense can do.
HdS84 3y ago
One Major Point for monorepos is the ability to eschew packages. In most languages creating, publishing and consuming a package is a lot of work, while in a monorepo you just add a reference to the code and are ready to go (except in react native...gaah that was pure horror). That's especially valuable if you need to refactor something and need to adjust it's dependencies. Doing that via packages is slow and painful. Via project references it's mich easier and has a tight feature loop, change+build+fix instead of of change+build+publish+consume+fix
- dmoy 3y ago
  Yup, agree completely. That's a natural extension of the same thing that enables atomic commits - suddenly just having direct library dependencies instead of packages isn't that big of a problem if you push everything into the monorepo. > That's especially valuable if you need to refactor something and need to adjust it's dependencies. And yes exactly, being able to change a library and all of its callers at the same time is pretty handy.