4 ms·

The index as a data structure is really starting to show its age, especially as developers adapt Git to monorepo scale. It's really fast for repositories up to

The index as a data structure is really starting to show its age, especially as developers adapt Git to monorepo scale. It's really fast for repositories up to a certain size, but big tech organizations grow exponentially, and start to suffer performance issues. At some point, you can't afford to use a data structure that scales with the size of the repo, and have to switch to one that scales with the size of the user's change. I spent a good chunk of time working around the lack of sparse indexes in libgit2, which produced speedups on the order of 500x for certain operations, because reading and writing the entire index is unnecessary for most users of a monorepo: https://github.com/libgit2/libgit2/issues/6036 https://github.com/libgit2/libgit2/issues/6036. I'm excited to see sparse indexes make their way into Git proper. Shameless plug: I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless https://github.com/arxanas/git-branchless, such as with in-memory rebases: https://blog.waleedkhan.name/in-memory-rebases/ https://blog.waleedkhan.name/in-memory-rebases/. Try it out if you work in a Git monorepo.

Buttersite 5y ago
Maybe this is a sign that monorepos are the problem?
- arxanas 5y ago
  Why would this indicate that monorepos are the problem?
  - chongli 5y ago
    It’s a philosophical conflict. Git was designed for distributed, decentralized version control and collaboration. Monorepos are inherently centralized. Thus git is the wrong tool for the job and all of this effort to adapt git to monorepo workflows runs counter to its philosophy.
    - status_quo69 5y ago
      I dunno, the linux kernel is pretty big and centralized by that definition, probably a monorepo by some definitions. I don't think this is a square peg round hole problem, I think that the operations that git must do coupled with the inevitable massive files that every company shoves into their repos (assets, binaries, you name it) makes git chug quite a bit.
    - arxanas 5y ago
      There are decentralized monorepos, such as gecko-dev (https://github.com/mozilla/gecko-dev https://github.com/mozilla/gecko-dev), which presumably has several forks in products like Iceweasel. I think the monorepo workflows which Git isn't good at are things like branching, code review, and feature/version management. But there's no reason that Git should have to be slow just because a repository is large. It's more things like "merge commits don't scale in a monorepo", which I would agree with, but that's not related to the performance of the index data structure.
    - smrq 5y ago
      I think you're conflating two different forms of "centralized". There is nothing inherently un-git about centralizing a bunch of code in a single repo; the decentralization that git is all about is that the repo state is decentralized across multiple machines.
stormbrew 5y ago
> I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless https://github.com/arxanas/git-branchless I'm intrigued by this but the readme could maybe use some work to describe how you envision it being used day-to-day? All the examples seem to be about using it to fix things but I'm not at all clear how it helps enable a new workflow. Even if it was just a link to a similar tool?
- arxanas 5y ago
  Thanks for the feedback. I also received this request today to document a relevant workflow: https://github.com/arxanas/git-branchless/issues/210 https://github.com/arxanas/git-branchless/issues/210. If you want to be notified when I write the documentation (hopefully today?), then you can watch that issue. There's a decent discussion here on "stacked changes": https://docs.graphite.dev/getting-started/why-use-stacked-changes https://docs.graphite.dev/getting-started/why-use-stacked-ch..., with references to other articles. This workflow is sometimes called development via "patch stack" or "stacked diffs". But that's just a part of the workflow which git-branchless enables. The most similar tool would be Mercurial as used at large companies (and in fact, `git-branchless` is, for now, just trying to get to feature parity with it). But I don't know if the feature set which engineers rely on is documented anywhere publicly. I use git-branchless 1) simply to scale to a monorepo, because `git move` is a lot faster than `git rebase`, and 2) to do highly speculative work and jump between many different approaches to the same problem (a kind of breadth-first search). I always had this problem with Git where I wanted to make many speculative changes, but branch and stash management got in the way. (For example, it's hard to update a commit which is a common ancestor of two or more branches. `git move` solves this.) The branchless workflow lets me be more nimble and update the commit graph more deftly, so that I can do experimental work much more easily.

Buttersite 5y ago

Maybe this is a sign that monorepos are the problem?

arxanas 5y ago
Why would this indicate that monorepos are the problem?
- chongli 5y ago
  It’s a philosophical conflict. Git was designed for distributed, decentralized version control and collaboration. Monorepos are inherently centralized. Thus git is the wrong tool for the job and all of this effort to adapt git to monorepo workflows runs counter to its philosophy.
  - status_quo69 5y ago
    I dunno, the linux kernel is pretty big and centralized by that definition, probably a monorepo by some definitions. I don't think this is a square peg round hole problem, I think that the operations that git must do coupled with the inevitable massive files that every company shoves into their repos (assets, binaries, you name it) makes git chug quite a bit.
  - arxanas 5y ago
    There are decentralized monorepos, such as gecko-dev (https://github.com/mozilla/gecko-dev https://github.com/mozilla/gecko-dev), which presumably has several forks in products like Iceweasel. I think the monorepo workflows which Git isn't good at are things like branching, code review, and feature/version management. But there's no reason that Git should have to be slow just because a repository is large. It's more things like "merge commits don't scale in a monorepo", which I would agree with, but that's not related to the performance of the index data structure.
  - smrq 5y ago
    I think you're conflating two different forms of "centralized". There is nothing inherently un-git about centralizing a bunch of code in a single repo; the decentralization that git is all about is that the repo state is decentralized across multiple machines.

stormbrew 5y ago

> I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless https://github.com/arxanas/git-branchless I'm intrigued by this but the readme could maybe use some work to describe how you envision it being used day-to-day? All the examples seem to be about using it to fix things but I'm not at all clear how it helps enable a new workflow. Even if it was just a link to a similar tool?

arxanas 5y ago
Thanks for the feedback. I also received this request today to document a relevant workflow: https://github.com/arxanas/git-branchless/issues/210 https://github.com/arxanas/git-branchless/issues/210. If you want to be notified when I write the documentation (hopefully today?), then you can watch that issue. There's a decent discussion here on "stacked changes": https://docs.graphite.dev/getting-started/why-use-stacked-changes https://docs.graphite.dev/getting-started/why-use-stacked-ch..., with references to other articles. This workflow is sometimes called development via "patch stack" or "stacked diffs". But that's just a part of the workflow which git-branchless enables. The most similar tool would be Mercurial as used at large companies (and in fact, `git-branchless` is, for now, just trying to get to feature parity with it). But I don't know if the feature set which engineers rely on is documented anywhere publicly. I use git-branchless 1) simply to scale to a monorepo, because `git move` is a lot faster than `git rebase`, and 2) to do highly speculative work and jump between many different approaches to the same problem (a kind of breadth-first search). I always had this problem with Git where I wanted to make many speculative changes, but branch and stash management got in the way. (For example, it's hard to update a commit which is a common ancestor of two or more branches. `git move` solves this.) The branchless workflow lets me be more nimble and update the commit graph more deftly, so that I can do experimental work much more easily.