3 ms·

It has nothing to do with the Windows filesystem; Git simply cannot support a 5 GB working tree on any filesystem. You can call this "pathological" but this th

It has nothing to do with the Windows filesystem; Git simply cannot support a 5 GB working tree on any filesystem. You can call this "pathological" but this throws a lot of shade on monorepos without much critical examination of how or when they might be useful.

espadrine 8y ago
To be precise, git's scaling issues mostly relate to centralization and file count rather than raw size. * Pushing changes to a central repo requires including upstream commits. With 1 commit/s to that central repo, all developers are stuck in a loop until their push succeeds. It is a human spinlock with high contention. * Some algorithms scale linearly with the number of server branches, such as pull-without-specifying-a-branch, which becomes too slow with 100K branches (a consequence of central repos). * Some algorithms are linear with the number of files, like git status. * Binary files don't compress nor deduplicate well, slowing pull and clone. Those issues apply to fossil and Mercurial. cf. https://docs.microsoft.com/en-us/azure/devops/learn/git/technical-scale-challenges https://docs.microsoft.com/en-us/azure/devops/learn/git/tech...
Chyzwar 8y ago
> It has nothing to do with the Windows filesystem; Git simply cannot support a 5 GB working tree on any filesystem. Can you provide a reference? I was searching a bit and only things I found was bugs in windows[1] for git lfs. > You can call this "pathological" but this throws a lot of shade on monorepos without much critical examination of how or when they might be useful. Windows codebase has 3.5 million files and its repo is 300GB in size. It is not normal. This is google or MS type of problem and not average git user. MS instead changing workflow decided to create GVFS[2] [1] https://github.com/git-lfs/git-lfs/issues/2434 https://github.com/git-lfs/git-lfs/issues/2434 [2] https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...
- ethomson 8y ago
  > Can you provide a reference? I was searching a bit and only things I found was bugs in windows[1] for git lfs. Apologies, I hastily mistyped, I meant 500 GB, not 5. (5 GB is about the size of my repository, which is not really so big at all and certainly something git can cope with on its own). This series of articles should illustrate some of the issues that VFS for Git tries to address. ("GVFS" is now called "VFS for Git".) https://docs.microsoft.com/en-us/azure/devops/learn/git/technical-scale-challenges https://docs.microsoft.com/en-us/azure/devops/learn/git/tech... And this is a series of articles from an engineer who's been working on improving perf in large repositories in general, not strictly related to the Windows repository: https://blogs.msdn.microsoft.com/devops/2018/06/25/supercharging-the-git-commit-graph/ https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar... > Windows codebase has 3.5 million files and its repo is 300GB in size. It is not normal. This is google or MS type of problem and not average git user. MS instead changing workflow decided to create GVFS[2] I didn't say it was normal. Indeed it's uncommon. I said it wasn't pathological.
  - qznc 8y ago
    Plastic SCM claims that 5TB works and that 50GB is the average size in their cloud offering. It seems that the Free Software world does not care about such use cases. https://medium.com/@psluaces/plasticscm-vs-git-c17934fad7ed https://medium.com/@psluaces/plasticscm-vs-git-c17934fad7ed
  - paulddraper 8y ago
    If you're versioning line-based text, and you have 80 ASCII characters per line, stored as ASCII/UTF-8, a worktree of 500GB has 6.7 billion lines. That's 3x the source line count of Google's entire monorepo. [1] So if you're using git for source code, 500GB is beyond pathological. If you're using git for other purposes, then yes you might need something like Annex/LFS/GVFS. [1] https://m-cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
fxfan 8y ago
Git lacking a functionality has nothing to do with monorepos not being useful.

espadrine 8y ago

To be precise, git's scaling issues mostly relate to centralization and file count rather than raw size. * Pushing changes to a central repo requires including upstream commits. With 1 commit/s to that central repo, all developers are stuck in a loop until their push succeeds. It is a human spinlock with high contention. * Some algorithms scale linearly with the number of server branches, such as pull-without-specifying-a-branch, which becomes too slow with 100K branches (a consequence of central repos). * Some algorithms are linear with the number of files, like git status. * Binary files don't compress nor deduplicate well, slowing pull and clone. Those issues apply to fossil and Mercurial. cf. https://docs.microsoft.com/en-us/azure/devops/learn/git/technical-scale-challenges https://docs.microsoft.com/en-us/azure/devops/learn/git/tech...

Chyzwar 8y ago

> It has nothing to do with the Windows filesystem; Git simply cannot support a 5 GB working tree on any filesystem. Can you provide a reference? I was searching a bit and only things I found was bugs in windows[1] for git lfs. > You can call this "pathological" but this throws a lot of shade on monorepos without much critical examination of how or when they might be useful. Windows codebase has 3.5 million files and its repo is 300GB in size. It is not normal. This is google or MS type of problem and not average git user. MS instead changing workflow decided to create GVFS[2] [1] https://github.com/git-lfs/git-lfs/issues/2434 https://github.com/git-lfs/git-lfs/issues/2434 [2] https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

ethomson 8y ago
> Can you provide a reference? I was searching a bit and only things I found was bugs in windows[1] for git lfs. Apologies, I hastily mistyped, I meant 500 GB, not 5. (5 GB is about the size of my repository, which is not really so big at all and certainly something git can cope with on its own). This series of articles should illustrate some of the issues that VFS for Git tries to address. ("GVFS" is now called "VFS for Git".) https://docs.microsoft.com/en-us/azure/devops/learn/git/technical-scale-challenges https://docs.microsoft.com/en-us/azure/devops/learn/git/tech... And this is a series of articles from an engineer who's been working on improving perf in large repositories in general, not strictly related to the Windows repository: https://blogs.msdn.microsoft.com/devops/2018/06/25/supercharging-the-git-commit-graph/ https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar... > Windows codebase has 3.5 million files and its repo is 300GB in size. It is not normal. This is google or MS type of problem and not average git user. MS instead changing workflow decided to create GVFS[2] I didn't say it was normal. Indeed it's uncommon. I said it wasn't pathological.
- qznc 8y ago
  Plastic SCM claims that 5TB works and that 50GB is the average size in their cloud offering. It seems that the Free Software world does not care about such use cases. https://medium.com/@psluaces/plasticscm-vs-git-c17934fad7ed https://medium.com/@psluaces/plasticscm-vs-git-c17934fad7ed
- paulddraper 8y ago
  If you're versioning line-based text, and you have 80 ASCII characters per line, stored as ASCII/UTF-8, a worktree of 500GB has 6.7 billion lines. That's 3x the source line count of Google's entire monorepo. [1] So if you're using git for source code, 500GB is beyond pathological. If you're using git for other purposes, then yes you might need something like Annex/LFS/GVFS. [1] https://m-cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...

fxfan 8y ago

Git lacking a functionality has nothing to do with monorepos not being useful.