6 ms·

On a philosophical note, git, being a distributed source control system has no limits to repo sizing, but github, being is a centralized "hub" suffers scaling p

On a philosophical note, git, being a distributed source control system has no limits to repo sizing, but github, being is a centralized "hub" suffers scaling problems. It's fascinating to watch these tradeoffs play out. Maybe, IPFS would come to the rescue one day.

cabirum 4y ago
The problem here is unbounded growth of a git repo. In this specific case, a size limit was triggered. In other circumstances, it would have required too much time to transfer or no storage space left. Anyway, the problem is that git stores all changes, forever. A better approach would be to clean old commits, or somehow merge them into snapshots of fixed timespans (like, anything older than a year get compressed into monthly changesets)
- mrguyorama 4y ago
  Git allows you to rewrite history, so you can "crush" old commits to reduce the size of history as needed.
  - cabirum 4y ago
    Sure, but rewriting it manually is a tedious process. Should be automated on github side, to keep repo size approx constant over time.
    - Arnavion 4y ago
      That would be silent data loss, so absolutely should not be automated.
    - blueflow 4y ago
      Github cant rewrite the refs on their own without breaking users stuff. They can only repack the existing objects, the squashing needs to be done by the developers. Also its a non-fast-forward thing, so it needs to be coordinated between the git users anyways.
    - saghm 4y ago
      I think this is what `git filter-branch` its supposed to be for: https://git-scm.com/docs/git-filter-branch https://git-scm.com/docs/git-filter-branch I've never used it before, but from what I understand, it's very powerful but also very confusing and easy to mess up, and of course with a sort of vague ambiguous name that makes it hard to discover; in other words, it's quintessentially git.
      - deleted 4y ago
        [deleted]
  - jacobr1 4y ago
    It would be great to have some kind of way to do this while still maintaining the merkle-tree.
    - jonhohle 4y ago
      Isn’t that what packs are for? The raw, content addressable object store has no inherent optimization for reducing repo size. Any changed file is completely copied until a higher level does something to compress that down.
- howinteresting 4y ago
  This is not a "problem", this is why source control exists. You should never rewrite published history.
  - cabirum 4y ago
    A value of a commit approaches zero as it gets older. After a certain threshold, no one will ever see it. Never say never; any reason why should we keep deadweight around?
    - howinteresting 4y ago
      Your premise is incorrect. The other day I was looking around a repository that's been through many migrations, and found a commit from 2004 that was relevant to my interests.
    - themerone 4y ago
      As long as a line code is in use, there is value in knowing who and when it was authored. If a 10 year old vulnerability is found in OpenSSL, it be nice to be able to investigate if it was an accident or an act of espionage.
  - tonnydourado 4y ago
    A more conservative approach would be some sort of layered storage/archiving, I guess. The older the commit, the less likely it is to be used, so it could be archive in a different storage, optimized for long term. This way you keep the "hot" history small, while keeping the full history still available.
    - howinteresting 4y ago
      That's generally how git packs are used at large organizations that host their own repositories. I'm sure GitHub does something similar.
  - crznp 4y ago
    I don't think that your points are actually in conflict. If this is my source code, I want the whole history. I want that 10-year old commit that isn't used in any current branch. A build machine may not need any history: it just wants to check out a particular branch as it is right now, and that works too. But there is an intermediate case: Let's say that I have an issue with a dependency. I might check out that code and want some history to know what has changed recently, but I don't need that huge zip file that was accidentally checked in and then removed 4 years ago. If it were a consistent problem, perhaps you'd invent some sort of 'shallow' or 'partial' clone, something like this: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...
Arnt 4y ago
I believe the problem here is related to LLVM using a CI model, where all changes eventually go to a single tree and tests are run on that single tree. Hosting that CI happen on another site than github wouldn't really change anything. Maybe change the limit, but all such CI hosts have a limit, for the same reason that all the search engines have a size limit for web pages they scrape.
tsimionescu 4y ago
I don't understand where git's distributed nature comes into play here. Perforce is fully centralized, and it also doesn't have any limits to repo size - as long as you have enough disk, P4 will handle it. In fact, Git's distributed nature actually makes it infamously bad at scaling with repo size - since it requires every "participant" in a repo to have a copy of the entire repo before they can start any work on it.
- dblitt 4y ago
  This is where something like VFSForGit [0] helps out. Instead of cloning the entire repo, it creates a virtual file system and fetches objects on demand. MSFT uses it internally for the Windows source tree (which now exceeds 300GB). [0]: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit
  - btown 4y ago
    VFSForGit seems to be in maintenance mode now - do you know what it was replaced by, and if there's something that works cross-platform?
    - AaronFriel 4y ago
      It was replaced by Scalar and is now merged into Git: Introduced: https://devblogs.microsoft.com/devops/introducing-scalar/ https://devblogs.microsoft.com/devops/introducing-scalar/ Integration into Git: https://github.blog/2022-10-13-the-story-of-scalar/ https://github.blog/2022-10-13-the-story-of-scalar/
- rvbissell 4y ago
  Shallow cloning of Git repos is a thing. Basically, you get a fake head commit (that includes all the files) and none of the real history. Useful if you only intend to build, or make changes locally. If you want to push, you have to unshallow first.
  - someguy101010 4y ago

cabirum 4y ago

The problem here is unbounded growth of a git repo. In this specific case, a size limit was triggered. In other circumstances, it would have required too much time to transfer or no storage space left. Anyway, the problem is that git stores all changes, forever. A better approach would be to clean old commits, or somehow merge them into snapshots of fixed timespans (like, anything older than a year get compressed into monthly changesets)

mrguyorama 4y ago
Git allows you to rewrite history, so you can "crush" old commits to reduce the size of history as needed.
- cabirum 4y ago
  Sure, but rewriting it manually is a tedious process. Should be automated on github side, to keep repo size approx constant over time.
  - Arnavion 4y ago
    That would be silent data loss, so absolutely should not be automated.
  - blueflow 4y ago
    Github cant rewrite the refs on their own without breaking users stuff. They can only repack the existing objects, the squashing needs to be done by the developers. Also its a non-fast-forward thing, so it needs to be coordinated between the git users anyways.
  - saghm 4y ago
    I think this is what `git filter-branch` its supposed to be for: https://git-scm.com/docs/git-filter-branch https://git-scm.com/docs/git-filter-branch I've never used it before, but from what I understand, it's very powerful but also very confusing and easy to mess up, and of course with a sort of vague ambiguous name that makes it hard to discover; in other words, it's quintessentially git.
    - deleted 4y ago
      [deleted]
- jacobr1 4y ago
  It would be great to have some kind of way to do this while still maintaining the merkle-tree.
  - jonhohle 4y ago
    Isn’t that what packs are for? The raw, content addressable object store has no inherent optimization for reducing repo size. Any changed file is completely copied until a higher level does something to compress that down.
howinteresting 4y ago
This is not a "problem", this is why source control exists. You should never rewrite published history.
- cabirum 4y ago
  A value of a commit approaches zero as it gets older. After a certain threshold, no one will ever see it. Never say never; any reason why should we keep deadweight around?
  - howinteresting 4y ago
    Your premise is incorrect. The other day I was looking around a repository that's been through many migrations, and found a commit from 2004 that was relevant to my interests.
  - themerone 4y ago
    As long as a line code is in use, there is value in knowing who and when it was authored. If a 10 year old vulnerability is found in OpenSSL, it be nice to be able to investigate if it was an accident or an act of espionage.
- tonnydourado 4y ago
  A more conservative approach would be some sort of layered storage/archiving, I guess. The older the commit, the less likely it is to be used, so it could be archive in a different storage, optimized for long term. This way you keep the "hot" history small, while keeping the full history still available.
  - howinteresting 4y ago
    That's generally how git packs are used at large organizations that host their own repositories. I'm sure GitHub does something similar.
- crznp 4y ago
  I don't think that your points are actually in conflict. If this is my source code, I want the whole history. I want that 10-year old commit that isn't used in any current branch. A build machine may not need any history: it just wants to check out a particular branch as it is right now, and that works too. But there is an intermediate case: Let's say that I have an issue with a dependency. I might check out that code and want some history to know what has changed recently, but I don't need that huge zip file that was accidentally checked in and then removed 4 years ago. If it were a consistent problem, perhaps you'd invent some sort of 'shallow' or 'partial' clone, something like this: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

Arnt 4y ago

I believe the problem here is related to LLVM using a CI model, where all changes eventually go to a single tree and tests are run on that single tree. Hosting that CI happen on another site than github wouldn't really change anything. Maybe change the limit, but all such CI hosts have a limit, for the same reason that all the search engines have a size limit for web pages they scrape.

tsimionescu 4y ago

I don't understand where git's distributed nature comes into play here. Perforce is fully centralized, and it also doesn't have any limits to repo size - as long as you have enough disk, P4 will handle it. In fact, Git's distributed nature actually makes it infamously bad at scaling with repo size - since it requires every "participant" in a repo to have a copy of the entire repo before they can start any work on it.

dblitt 4y ago
This is where something like VFSForGit [0] helps out. Instead of cloning the entire repo, it creates a virtual file system and fetches objects on demand. MSFT uses it internally for the Windows source tree (which now exceeds 300GB). [0]: https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit
- btown 4y ago
  VFSForGit seems to be in maintenance mode now - do you know what it was replaced by, and if there's something that works cross-platform?
  - AaronFriel 4y ago
    It was replaced by Scalar and is now merged into Git: Introduced: https://devblogs.microsoft.com/devops/introducing-scalar/ https://devblogs.microsoft.com/devops/introducing-scalar/ Integration into Git: https://github.blog/2022-10-13-the-story-of-scalar/ https://github.blog/2022-10-13-the-story-of-scalar/
rvbissell 4y ago
Shallow cloning of Git repos is a thing. Basically, you get a fake head commit (that includes all the files) and none of the real history. Useful if you only intend to build, or make changes locally. If you want to push, you have to unshallow first.
- someguy101010 4y ago