10 ms·

Git's biggest flaw is that it doesn't scale. If a new system can fix that without sacrificing any of Git's benefits, I think it can topple Git. It's ironic th

by comex 7y ago

Git's biggest flaw is that it doesn't scale. If a new system can fix that without sacrificing any of Git's benefits, I think it can topple Git. It's ironic that Git was popularized in the same era as monorepos, yet Git is a poor fit for monorepos. There have been some attempts to work around this. Google's `repo` command is a wrapper around Git that treats a set of smaller repos like one big one, but it's a (very) leaky abstraction. Microsoft's GVFS is a promising attempt to truly scale Git to giant repos, but it's developed as an addon rather than a core part of Git, and so far it only works on Windows (with macOS support in development). GVFS arguably has the potential to become an ubiquitous part of the Git experience, someday... but it probably won't. Git also has trouble with large files. The situation is better these days, as most people have seemingly standardized on git-lfs (over its older competitor git-annex), and it works pretty well. Nevertheless, it feels like a hack that "large" files have to be managed using a completely different system from normal files, one which (again) is not a core part of Git. There exist version control systems that do scale well to large repos and large files, but all the ones I've heard of have other disadvantages compared to Git. For example, they're not decentralized, or they're not as lightning-fast as Git is in smaller repos, or they're harder to use. That's why I think there's room for a future competitor! (Fossil is not that competitor. From what I've heard, it neither scales well nor matches Git in performance for small repos, unfortunately.)

james_s_tayler 7y ago
I'd say the number of git repos on Earth that would encounter problems of that nature would be a vanishingly microscopic minority. Sure, it's a problem for those companies but it's not a problem for anyone else.
- hamandcheese 7y ago
  Vanishingly small in number, but quite significant in terms of the number of developers working in them.
  - Polylactic_acid 7y ago
    All of the organizations that have outgrown git will have such incredibly specific requirements meaning nothing but a custom built tool will work for them.
    - rumanator 7y ago
      The so called problem would also vanish if the monorepo was modularized and broken up into smaller repos.
      - adrianN 7y ago
        That's probably a harder task for an existing monorepo that is too big for git than writing a replacement for git that works with a repo of that size.
        rumanator 7y ago
        The problem boils down to refactoring a large monolith. I feel like Git is a scapegoat for a much larger problem.
        neerajsi 7y ago
        Let's say you started with a well factored set of code that is managed within your organization. What advantage is there to having multiple repos if you're not limited by your tools? Refactoring is easier within a single repo...
        neoeldex 7y ago
        There probably will be a plethora of different hard issues to fix in such situations. It's also easier to institute change in a dictatorship as opposed to a democracy (being a dictator that is :).
        adrianN 7y ago
- TheFuntastic 7y ago
  This reads to me as a failure of imagination. Any mid size game development shop is going to feel this pain - not just giants like Microsoft and Google. I believe the Unity Game Engine has a user base in the millions? Even a subset of that may be small in comparison to the entire developer population but by no means microscopic.
kemitche 7y ago
I disagree that Git's biggest flaw is its lack of scalability. Cases where git needs to scale tend to be isolated to companies that have the manpower to build a finely-tuned replacement (see: MS, Google). Git's flaws are primarily in usability/UX. But I think for its purpose, functionality is far more important than a perfect UX. I'm perfectly happy knowing I might have to Google how to do something in Git as long as I can feel confident that Git will have the power to do whatever it is I'm trying to do. A competitor would need to do what git does as well as git does it, with a UX that is not just marginally better but categorically better, to unseat git. (Marginally better isn't strong enough to overcome incumbent use cases) And for the record: I think git-lfs issues are primarily usability issues, and tech improvements. The tech enhancements will be solved if there's enough desire, and as I mentioned the usability problems are more annoyances than actual problems.
- rtpg 7y ago
  Scaling isn't even just about number of files or size of them. A problem I've hit is just in having cross-repo stuff work well. Monorepos are helpful partly because git submodules are not ideal for good workflows, and splitting stuff across multiple git repos can backfire (it doesn't help that almost all the tooling around CI and the like is repo-based instead of project based). I would love a layer over Git to handle workflow issues related to multi-repo projects
  - nsomaru 7y ago
    Sub modules have terrible UX and are completely counterintuitive. Subtrees work well for my small repos when I need to vendor something in
    - rossmohax 7y ago
      There is also a git subrepo from subtree author, worth checking out IMHO
- MaulingMonkey 7y ago
  > Cases where git needs to scale tend to be isolated to companies that have the manpower to build a finely-tuned replacement (see: MS, Google). I constantly run into git scalability issues as an individual. I don't use any of the UI clients because they all fail hard on mostly-code git repositories. I abandoned my VisualRust port in part because the mere 100MB of mingw binaries involved for that meant it was using github LFS, which meant CI was hitting github quota limits, and as I wasn't part of the organization - nevermind an admin with billing rights - I couldn't even pay to up said quota limits paying out of pocket myself even if I wanted to. I'm not going to inflict git's command line experience - which confounds and confuses even seasoned programmers - on any of the less technical artists that might be employed at a typical gamedev shop, even if git might be able to scale acceptably if locally self-hosted at a single-digit employee shop. A few dozen or hundred employees? Forget it. Use perforce, even though it costs $$$, is far from perfect, and also has plenty of scaling issues eventually.
dr01d 7y ago
What are you smoking? GitHub and gitlab together basically encompass almost ALL of humanities efforts with regards to software development. Doesn't scale??? You are seriously delusional.
yaih 7y ago
"Barney Oliver was a good man. He wrote a letter one time to the IEEE. At that time the official shelf space at Bell Labs was so much and the height of the IEEE Proceedings at that time was larger; and since you couldn't change the size of the official shelf space he wrote this letter to the IEEE Publication person saying, since so many IEEE members were at Bell Labs and since the official space was so high the journal size should be changed." - http://www.paulgraham.com/hamming.html http://www.paulgraham.com/hamming.html
- comex 7y ago
  What is the analogy here? The first guess that came to my mind was the monorepo versus multi-repo debate: since Git can only support repos that are so large (shelf space) without getting slow, you should split up your repos (journal size) even if semantically you would prefer a monorepo. But that would support the point I was making, whereas the obliqueness of your reply makes me think you probably meant to criticize it.
  - ric2b 7y ago
    I think he's comparing the journal to the tool (harder to change, impacts everyone) and the shelf to the problem that only impacts a few organizations but actually a lot of people because those organizations are so large.
    - comex 7y ago
      I guess that makes sense. But if that's the analogy, there's a significant difference between the situations. In that example, there was nothing inherently wrong with the journal's size, other than it not matching Bell Labs' arbitrary choice of shelf layout. Git, on the other hand, would be inherently a better tool if it had better performance on large repos (without sacrificing its suitability for small repos).
kccqzy 7y ago
Mercurial is probably that competitor. Only slightly slower than Git. Works on very large monorepos (as large as Facebook's or Google's monorepo). Very similar workflow as compared to Git, with some minor differences in terminology.
- reissbaker 7y ago
  As an FB employee, I use hg regularly (because it is required). I would not use it as a git replacement for non-FB-sized repos. It has some weird design choices (e.g. branching is bad), and it very often requires manual intervention for merges that git performs correctly and automatically. You can get around branching-is-bad by changing your workflows a bit, but you can't get around the bad merges: over time it's like death by a thousand papercuts.
  - hvidgaard 7y ago
    What kind of merges does git handle that hg doesn't? If it's just a matter of figuring what goes where, someone that uses hg daily could copy the implementation from hg. It could be a big organization that uses daily for instance.
  - kccqzy 7y ago
    Paper cuts can be addressed with more users reporting bugs and contributing fixes. The fundamental design issues with git that prevent scalability cannot. As a Google employee I use hg every day, even though it's not required. (Some teams at Google do mandate its use, but these are few and far between.) I don't use branches, but I use bookmarks. I didn't notice any merges that really ought to be performed automatically but were not; in any case I use Meld to resolve merge conflicts and it's easy enough to do occasionally.
    - reissbaker 7y ago
      For most people, avoiding thousands of papercuts is better than scaling massively. Few people need massive scale, but everyone hates papercuts. I'm also not certain that the "fundamental design issues" with git are truly fundamental to its design. For example, partial clones and sparse checkouts are seeing increasing support in recent versions of git — and those are really all you need.
squiggleblaz 7y ago
The only way I've ever successfully `git clone`d my work repo is from another locally connected device. Even with shallow and then gradually unshallowing it, it will not generally complete before the internet falls over. Nowadays, a new computer means a git clone (or just plain copy-paste) of a USB stick from the old one. This seems like it's a single feature of git that could be written, but if you told me "there's something that works better for large, twenty year old repos", I'd probably take that. I don't know how Linux survives, but maybe it's just that you only rarely git clone your large repos. (Or maybe it's just that intercontinental internet is less reliable than intracontinental, so that if you're in the US it's a non issue.)
- orf 7y ago
  I'm guessing your work repo is has lots of large binary files in it's history?
mikl 7y ago
Git scales well enough for almost everyone (especially if you have a little discipline with what you put in the repo). It’s only huge megacorps that need larger scale things like GVFS. As for large files, that is not what Git is for. Git is for source code. Much like how you don’t put large files in your RDBMS, you should not be putting them in your SCM either.
- int_19h 7y ago
  What if you need to version them? Git imposes a very specific versioning model: version is a property of the entire repository. Thus, not including some file in the repo implies that it's not versioned in the same manner. It's not just a function of binary vs source.
  - mikl 7y ago
    Versioning big binary blobs is not what Git was designed for. It’ll do fine with smaller assets like icons and the like, but its data model is based on everyone using the repo having a local copy of the full repository history. You can’t easily purge old data. That scales poorly if you want to use it for audio/video files or other large data sets. You can still do it if you want, but you might be better served using https://git-lfs.github.com/ https://git-lfs.github.com/ or using another system designed for that purpose.
- P_I_Staker 7y ago
  Honestly, you can use Git for large files with lfs. I wouldn't say I love this approach, but it isn't that bad now. You do have to make room for yet-another-tool, and you now have centralized version control comingling with your distributed tool (essentially making it central); but you can still use everything you love about git, and if your lfs doesn't change, you don't need to be connected to a server. It certainly feels pretty absurd. This isn't even a problem in SVN, but now we're tacking on another tool that you have to learn, and introduces issues.
qznc 7y ago
I believe there will be no scalable open-source VCS because the incentives are not there. While the technical problem is interesting, I decided not to work on it because of this. http://beza1e1.tuxen.de/monorepo_vcs.html http://beza1e1.tuxen.de/monorepo_vcs.html
fuckface123 7y ago
Monorepos are a terrible idea.
shurcooL 7y ago
> Google's `repo` command is a wrapper around Git that treats a set of smaller repos like one big one, but it's a (very) leaky abstraction. Could you please provide a link to it? I’m very interested in seeing this command, but ironically it’s not a name that’s easy to google for. Edit: I was very wrong, searching for “google repo command” displayed https://gerrit.googlesource.com/git-repo https://gerrit.googlesource.com/git-repo as the very first result.

by comex 7y ago

james_s_tayler 7y ago

I'd say the number of git repos on Earth that would encounter problems of that nature would be a vanishingly microscopic minority. Sure, it's a problem for those companies but it's not a problem for anyone else.

hamandcheese 7y ago
Vanishingly small in number, but quite significant in terms of the number of developers working in them.
- Polylactic_acid 7y ago
  All of the organizations that have outgrown git will have such incredibly specific requirements meaning nothing but a custom built tool will work for them.
  - rumanator 7y ago
    The so called problem would also vanish if the monorepo was modularized and broken up into smaller repos.
    - adrianN 7y ago
      That's probably a harder task for an existing monorepo that is too big for git than writing a replacement for git that works with a repo of that size.
      - rumanator 7y ago
        The problem boils down to refactoring a large monolith. I feel like Git is a scapegoat for a much larger problem.
        neerajsi 7y ago
        Let's say you started with a well factored set of code that is managed within your organization. What advantage is there to having multiple repos if you're not limited by your tools? Refactoring is easier within a single repo...
        neoeldex 7y ago
        There probably will be a plethora of different hard issues to fix in such situations. It's also easier to institute change in a dictatorship as opposed to a democracy (being a dictator that is :).
        adrianN 7y ago
TheFuntastic 7y ago
This reads to me as a failure of imagination. Any mid size game development shop is going to feel this pain - not just giants like Microsoft and Google. I believe the Unity Game Engine has a user base in the millions? Even a subset of that may be small in comparison to the entire developer population but by no means microscopic.

kemitche 7y ago

I disagree that Git's biggest flaw is its lack of scalability. Cases where git needs to scale tend to be isolated to companies that have the manpower to build a finely-tuned replacement (see: MS, Google). Git's flaws are primarily in usability/UX. But I think for its purpose, functionality is far more important than a perfect UX. I'm perfectly happy knowing I might have to Google how to do something in Git as long as I can feel confident that Git will have the power to do whatever it is I'm trying to do. A competitor would need to do what git does as well as git does it, with a UX that is not just marginally better but categorically better, to unseat git. (Marginally better isn't strong enough to overcome incumbent use cases) And for the record: I think git-lfs issues are primarily usability issues, and tech improvements. The tech enhancements will be solved if there's enough desire, and as I mentioned the usability problems are more annoyances than actual problems.

rtpg 7y ago
Scaling isn't even just about number of files or size of them. A problem I've hit is just in having cross-repo stuff work well. Monorepos are helpful partly because git submodules are not ideal for good workflows, and splitting stuff across multiple git repos can backfire (it doesn't help that almost all the tooling around CI and the like is repo-based instead of project based). I would love a layer over Git to handle workflow issues related to multi-repo projects
- nsomaru 7y ago
  Sub modules have terrible UX and are completely counterintuitive. Subtrees work well for my small repos when I need to vendor something in
  - rossmohax 7y ago
    There is also a git subrepo from subtree author, worth checking out IMHO
MaulingMonkey 7y ago
> Cases where git needs to scale tend to be isolated to companies that have the manpower to build a finely-tuned replacement (see: MS, Google). I constantly run into git scalability issues as an individual. I don't use any of the UI clients because they all fail hard on mostly-code git repositories. I abandoned my VisualRust port in part because the mere 100MB of mingw binaries involved for that meant it was using github LFS, which meant CI was hitting github quota limits, and as I wasn't part of the organization - nevermind an admin with billing rights - I couldn't even pay to up said quota limits paying out of pocket myself even if I wanted to. I'm not going to inflict git's command line experience - which confounds and confuses even seasoned programmers - on any of the less technical artists that might be employed at a typical gamedev shop, even if git might be able to scale acceptably if locally self-hosted at a single-digit employee shop. A few dozen or hundred employees? Forget it. Use perforce, even though it costs $$$, is far from perfect, and also has plenty of scaling issues eventually.

dr01d 7y ago

What are you smoking? GitHub and gitlab together basically encompass almost ALL of humanities efforts with regards to software development. Doesn't scale??? You are seriously delusional.

yaih 7y ago

"Barney Oliver was a good man. He wrote a letter one time to the IEEE. At that time the official shelf space at Bell Labs was so much and the height of the IEEE Proceedings at that time was larger; and since you couldn't change the size of the official shelf space he wrote this letter to the IEEE Publication person saying, since so many IEEE members were at Bell Labs and since the official space was so high the journal size should be changed." - http://www.paulgraham.com/hamming.html http://www.paulgraham.com/hamming.html

comex 7y ago
What is the analogy here? The first guess that came to my mind was the monorepo versus multi-repo debate: since Git can only support repos that are so large (shelf space) without getting slow, you should split up your repos (journal size) even if semantically you would prefer a monorepo. But that would support the point I was making, whereas the obliqueness of your reply makes me think you probably meant to criticize it.
- ric2b 7y ago
  I think he's comparing the journal to the tool (harder to change, impacts everyone) and the shelf to the problem that only impacts a few organizations but actually a lot of people because those organizations are so large.
  - comex 7y ago
    I guess that makes sense. But if that's the analogy, there's a significant difference between the situations. In that example, there was nothing inherently wrong with the journal's size, other than it not matching Bell Labs' arbitrary choice of shelf layout. Git, on the other hand, would be inherently a better tool if it had better performance on large repos (without sacrificing its suitability for small repos).

kccqzy 7y ago

Mercurial is probably that competitor. Only slightly slower than Git. Works on very large monorepos (as large as Facebook's or Google's monorepo). Very similar workflow as compared to Git, with some minor differences in terminology.

reissbaker 7y ago
As an FB employee, I use hg regularly (because it is required). I would not use it as a git replacement for non-FB-sized repos. It has some weird design choices (e.g. branching is bad), and it very often requires manual intervention for merges that git performs correctly and automatically. You can get around branching-is-bad by changing your workflows a bit, but you can't get around the bad merges: over time it's like death by a thousand papercuts.
- hvidgaard 7y ago
  What kind of merges does git handle that hg doesn't? If it's just a matter of figuring what goes where, someone that uses hg daily could copy the implementation from hg. It could be a big organization that uses daily for instance.
- kccqzy 7y ago
  Paper cuts can be addressed with more users reporting bugs and contributing fixes. The fundamental design issues with git that prevent scalability cannot. As a Google employee I use hg every day, even though it's not required. (Some teams at Google do mandate its use, but these are few and far between.) I don't use branches, but I use bookmarks. I didn't notice any merges that really ought to be performed automatically but were not; in any case I use Meld to resolve merge conflicts and it's easy enough to do occasionally.
  - reissbaker 7y ago
    For most people, avoiding thousands of papercuts is better than scaling massively. Few people need massive scale, but everyone hates papercuts. I'm also not certain that the "fundamental design issues" with git are truly fundamental to its design. For example, partial clones and sparse checkouts are seeing increasing support in recent versions of git — and those are really all you need.

squiggleblaz 7y ago

The only way I've ever successfully `git clone`d my work repo is from another locally connected device. Even with shallow and then gradually unshallowing it, it will not generally complete before the internet falls over. Nowadays, a new computer means a git clone (or just plain copy-paste) of a USB stick from the old one. This seems like it's a single feature of git that could be written, but if you told me "there's something that works better for large, twenty year old repos", I'd probably take that. I don't know how Linux survives, but maybe it's just that you only rarely git clone your large repos. (Or maybe it's just that intercontinental internet is less reliable than intracontinental, so that if you're in the US it's a non issue.)

orf 7y ago
I'm guessing your work repo is has lots of large binary files in it's history?

mikl 7y ago

Git scales well enough for almost everyone (especially if you have a little discipline with what you put in the repo). It’s only huge megacorps that need larger scale things like GVFS. As for large files, that is not what Git is for. Git is for source code. Much like how you don’t put large files in your RDBMS, you should not be putting them in your SCM either.

int_19h 7y ago
What if you need to version them? Git imposes a very specific versioning model: version is a property of the entire repository. Thus, not including some file in the repo implies that it's not versioned in the same manner. It's not just a function of binary vs source.
- mikl 7y ago
  Versioning big binary blobs is not what Git was designed for. It’ll do fine with smaller assets like icons and the like, but its data model is based on everyone using the repo having a local copy of the full repository history. You can’t easily purge old data. That scales poorly if you want to use it for audio/video files or other large data sets. You can still do it if you want, but you might be better served using https://git-lfs.github.com/ https://git-lfs.github.com/ or using another system designed for that purpose.
P_I_Staker 7y ago
Honestly, you can use Git for large files with lfs. I wouldn't say I love this approach, but it isn't that bad now. You do have to make room for yet-another-tool, and you now have centralized version control comingling with your distributed tool (essentially making it central); but you can still use everything you love about git, and if your lfs doesn't change, you don't need to be connected to a server. It certainly feels pretty absurd. This isn't even a problem in SVN, but now we're tacking on another tool that you have to learn, and introduces issues.

qznc 7y ago

I believe there will be no scalable open-source VCS because the incentives are not there. While the technical problem is interesting, I decided not to work on it because of this. http://beza1e1.tuxen.de/monorepo_vcs.html http://beza1e1.tuxen.de/monorepo_vcs.html

fuckface123 7y ago

Monorepos are a terrible idea.

shurcooL 7y ago

> Google's `repo` command is a wrapper around Git that treats a set of smaller repos like one big one, but it's a (very) leaky abstraction. Could you please provide a link to it? I’m very interested in seeing this command, but ironically it’s not a name that’s easy to google for. Edit: I was very wrong, searching for “google repo command” displayed https://gerrit.googlesource.com/git-repo https://gerrit.googlesource.com/git-repo as the very first result.