4 ms·

Github uses their own storage backend which I believe shares objects across all of projects regardless of whether they are explicit forks or not.

• zbentley 9y ago
  That's really neat. Is there any documentation/discussion available on that technology? It sounds like something that would be fascinating to learn about.
  • Edmond 9y ago
    I am not sure if the parent's claim is true, ie that Github is storing objects and sharing them across forked repos. If they are, then it likely just a direct implementation of git the technology. you can see how git stores data here: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
    • aidos 9y ago
      Not an expert, but that would only work if there was a single repo, right?
      • Edmond 9y ago
        Not an expert either :) There is the concept of submodules which allows for multiple repos while maintaining the checksum mechanics that allows sharing the same bit of information between branches and across commits: https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules The trick is that git maintains an abstract file system (ie a graph) across commits. The graph consist of pointers to content without having to create a clone of the actual content for every new version of said graph....it gets a little dizzy to explain but it is really not too complicated :)
      • allenz 9y ago
        Not necessarily. To share blobs, GitHub would need to replace the standard git filesystem blob storage with a distributed database of blobs. All Git repos would share this distributed database.
  • rrix2 9y ago
    This talk[1] from git merge has a lot of information about how they run their backend storage and what silly things can happen as a result of that. [1] https://www.youtube.com/watch?v=-ZNKR9wFe8o https://www.youtube.com/watch?v=-ZNKR9wFe8o
• ethomson 9y ago
  GitHub only shares objects among forks. Source: I used to work on GitHub's Git Infrastructure team, but this is publicly available information. You can read about their architecture in their discussions of Spokes, which replicates repository networks (the original repository and its forks) across data centers. eg: https://githubengineering.com/stretching-spokes/ https://githubengineering.com/stretching-spokes/ Trying to put all of GitHub's object files in a single packfile - even just putting them on a single server - would be impossible. But even on hosting providers with a bespoke implementation - that do not use core git to manage Git repositories - this would be challenging. We have a custom Git server implementation in Visual Studio Team Services, but it still makes sense to shard object storage with the repository: you have to worry about scalability and performance, but also things like data sovereignty. We can't just put a user's git repository in some global SQL Azure database that contains all the repositories in Visual Studio Team Services: the repositories need to be geographically located with the VSTS account they created.
  • dfox 9y ago
    I vaguely remember that somebody from github claimed in HN comment that all objects are shared. Puting whole github in single packfile is obviously impractical, but having whole github on some bespoke Venti/IPFS-style content addressable object store is not.
    • ethomson 9y ago
      It's not impossible but it too is impractical for performance reasons. Providers that run git core repack repositories regularly because packfiles are efficient; loose objects are horribly inefficient - even on the local filesystem. Moving them to a network filesystem is impractical.