9 ms·

I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI. I would have expected that the repos were br

I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI. I would have expected that the repos were broken up into individual objects and stored in an object store. This should make pushes much faster as you have basically infinitely scalable writes. However it does make pulls more difficult. However computing packfiles could still be done (asynchronously) and with some attention to data-locality it should be possible. This would be a huge rewrite of some internals but seems like it would be a lot easier to manage. It would also provide a some benefits as objects could be shared between repos (although some care would probably be necessary for hash collisions) and it would remove some of the oddness about forks (as IIUC they effectively share the repo with the "parent" repo). I would love to know if something like this has been considered and why the decided against it.

Denvercoder9 6y ago
> This should make pushes much faster as you have basically infinitely scalable writes. However it does make pulls more difficult. I bet GitHub has much more read traffic than write traffic, so this trade-off does not make sense.
- random5634 6y ago
  Seriously, imagine the compute and requests costs to assemble a large git pull.
  - cordite 6y ago
    Sounds a lot like this here https://github.com/Homebrew/brew/pull/9383 https://github.com/Homebrew/brew/pull/9383
- kevincox 6y ago
  I said "difficult" not expensive. Once you assembled the packfiles (much like they do today) it should be roughly the same cost.
- WorldMaker 6y ago
  Yes and I would also imagine that the trade-offs between writing a proprietary object storage and reusing the battle-tested object storage that everyone else uses would have been considered as well. It seems like the sort of thing that would be an interesting open source research topic if you could build an object database for git that performs better than its packed in filesystem object store. But it's probably not something you want to do as a proprietary project with fewer eyeballs on its performance trade-offs and more engineering work every time git slightly changes its object storage behavior which would remain tuned for the filesystem object store because it was entirely unaware of your efforts.
parhamn 6y ago
Have you ever tried it? It's not remotely performant and wouldn't make sense since GH is read heavy. Plus I'm sure they spend a lot of time thinking about this stuff, no? If you want to get your feet wet, check out go-git[1]. It's a native golang implementation of git. They have a storage layer abstracted over a lean interface that you quickly create alternative drivers for in golang. You'll be effectively implementing poorly sharded file system on a database, then it becomes obvious why scaling the FS is just easier. [1] https://github.com/go-git/go-git/tree/master/storage https://github.com/go-git/go-git/tree/master/storage
- brown9-2 6y ago
  > Plus I'm sure they spend a lot of time thinking about this stuff, no? I think this is unfair - the author was not insinuating that the people who designed this system at Github are stupid in some way, but just asking if other architectures have been considered.
  - parhamn 6y ago
    > ...Github are stupid in some way, but just asking if other architectures have been considered. To me, asking an engineering org if they’ve considered alternative architectures for their main engineering problem is silly at best, overconfident at worst.
    - ben0x539 6y ago
      I think the main point of the comment was asking why they decided against it. At one point the wording mildly suggests the possibility that no one at github has thought about it: > I would love to know if something like this has been considered and why the decided against it. ... but that still sounds more like a grammatical hedge than an actual suggestion that github didn't think it through. imo it's fair to lay out why you're surprised about some decision in the hopes that someone will enlighten you, even if it can be tricky to phrase that without coming off like a "why didn't you just..." comment.
    - brown9-2 6y ago
Ericson2314 6y ago
I always had the impression GitHub was not preemptively investing in the fundamentals like that. So yeah, agree it's bummer but also not surprised. And hey, at least that means a post GitHub FOSS world won't be leaving fundamental improvements behind!
lumost 6y ago
I am not a github employee, but my 2 cents. An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS's can be distributed to arbitrary write velocity given an appropriately distributed block storage solution ( which will provide the k/v API of an object store that you're looking for ). Distributed FS's are conveniently compatible with most POSIX operations rather than requiring bespoke integration. Most object stores are optimized for largish objects and lack the ability to condense records into an individual write (via the block API) or pre-emptively pre-fetch the most likely next set of requested blocks. In the GitHub's case the choice of diverging from GitCLI/FS based storage APIs could lead to long term support issues and an implicit "github" flavor of git rather than improving the core git toolchain. Object Stores are great, but if you need some form of index they get slow and painful really fast.
- ddorian43 6y ago
  You should be able to split the object store into 2 systems: 1 metadata (think rdbms/nosql/etc) and a blob-data service, keeping large files, think 10KB+. Both systems should be able to be more efficient than the current method. Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc. But somebody has to create it, that's the issue.
  - lumost 6y ago
    That's exactly how distributed filesystems are built. Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.
- merb 6y ago
  https://github.com/juicedata/juicefs https://github.com/juicedata/juicefs has an index implementation and is backed up by object storage.
  - 6y ago
hobofan 6y ago
> I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI. Maybe I'm a bit dense, but how did you get that from the article? I'm fairly certain that in other pieces of writing they showed that they are using an object store, and I'm guessing that's what the "file servers" in the article are.
- stuhood 6y ago
  `git repack` is an operation that is fairly specific to git's default file format: if they were storing objects in any (other) database, it is very unlikely that they would experience blocking repack operations, as that is an area where databases are highly optimized to execute incrementally.
oconnor663 6y ago
I think the GitHub folks have written more than one article about this. I'm not sure I can find the one I'm thinking of, but here's another one: https://github.blog/2016-04-05-introducing-dgit/ https://github.blog/2016-04-05-introducing-dgit/ > Perhaps it’s surprising that GitHub’s repository-storage tier, DGit, is built using the same technologies. Why not a SAN? A distributed file system? Some other magical cloud technology that abstracts away the problem of storing bits durably? The answer is simple: it’s fast and it’s robust.
mvzvm 6y ago
This kind of comment is why every single project needs to be justified with "What problems are you solving?" and "What usecase are you supporting?". Because I could 150% imagine somebody getting excited about this and then: 1) Framing it as such with poor justification "a lot easier to manage" 2) "This would be a huge rewrite of some internals" Becoming a multi-year migration quagmire 3) The dawning realization that you have used a write-heavy architecture in a read-heavy system
- saurik 6y ago
  I actually implemented this one day--storing all of the git objects into a PostgreSQL database, as a modification to the normal C program copy of git (not some go fork or "libgit2" or whatever people seem to expect you have to use)--and it was really easy: git already is designed to do its object access via a simple internal abstraction, so it was a trivial tap.
  - jhatemyjob 6y ago
    Why postgres? Wouldn't SQLite be better?
wmf 6y ago
Vincent Marti's legendary and funny "My Mom Told Me That git Doesn't Scale" presentation explains why GitHub chose filesystems instead of something fancier. https://www.youtube.com/watch?v=Ri8hSZNKzu4 https://www.youtube.com/watch?v=Ri8hSZNKzu4
- iudqnolq 6y ago
  Thanks, I hadn't seen that before. I really liked the line "People think we're a Ruby shop, but actually we're a UNIX shop. Everything else is an implementation detail".
remram 6y ago
> infinitely scalable writes Scalable receive, but having to send hundred of thousands of objects one-by-one is very inefficient. The fact that there are hundreds of thousands of servers to receive them does not help you. (and same problem when you pull)
gh-throw 6y ago
Not to pile on (as the idea isn't necessarily, immediately crazy): 1) The filesystem is a surprisingly decent datastore, and, 2) Unless your write-volume is unreasonably high for a GH repo, your main problem is making sure your locking (and unlocking) is rock-fucking-solid; Git has built-in locking, but I bet Github has either replaced it or has some serious state-machine stuff going on to manage it outside of Git itself, also and, 3) Reads on files are ultra-cacheable, obviously, and reads on HTML pages are too, and serving a second-or-two-stale read isn't a problem ~100% of the time in GH's use case, so read volume isn't really a problem. [EDIT] source: have designed and developed a git-backed, client-accessible product at a much smaller scale than GH, but enough to see where the problems & strengths could/would be, at greater scale. [EDIT EDIT] oh and with a bunch of clients using a bunch of different repos that effectively never interact except through normal git-push/PR behavior, obviously horizontal scaling is a breeze and a half. All you need is a little routing to figure out which repo is where. Git itself, plus your locking solution (whatever that may be), give you the tools you need to migrate from one server to another if you need to move a repo, probably with an unnoticeable amount of downtime (=locked repo) if you're clever about it.

Denvercoder9 6y ago

> This should make pushes much faster as you have basically infinitely scalable writes. However it does make pulls more difficult. I bet GitHub has much more read traffic than write traffic, so this trade-off does not make sense.

random5634 6y ago
Seriously, imagine the compute and requests costs to assemble a large git pull.
- cordite 6y ago
  Sounds a lot like this here https://github.com/Homebrew/brew/pull/9383 https://github.com/Homebrew/brew/pull/9383
kevincox 6y ago
I said "difficult" not expensive. Once you assembled the packfiles (much like they do today) it should be roughly the same cost.
WorldMaker 6y ago
Yes and I would also imagine that the trade-offs between writing a proprietary object storage and reusing the battle-tested object storage that everyone else uses would have been considered as well. It seems like the sort of thing that would be an interesting open source research topic if you could build an object database for git that performs better than its packed in filesystem object store. But it's probably not something you want to do as a proprietary project with fewer eyeballs on its performance trade-offs and more engineering work every time git slightly changes its object storage behavior which would remain tuned for the filesystem object store because it was entirely unaware of your efforts.

parhamn 6y ago

Have you ever tried it? It's not remotely performant and wouldn't make sense since GH is read heavy. Plus I'm sure they spend a lot of time thinking about this stuff, no? If you want to get your feet wet, check out go-git[1]. It's a native golang implementation of git. They have a storage layer abstracted over a lean interface that you quickly create alternative drivers for in golang. You'll be effectively implementing poorly sharded file system on a database, then it becomes obvious why scaling the FS is just easier. [1] https://github.com/go-git/go-git/tree/master/storage https://github.com/go-git/go-git/tree/master/storage

brown9-2 6y ago
> Plus I'm sure they spend a lot of time thinking about this stuff, no? I think this is unfair - the author was not insinuating that the people who designed this system at Github are stupid in some way, but just asking if other architectures have been considered.
- parhamn 6y ago
  > ...Github are stupid in some way, but just asking if other architectures have been considered. To me, asking an engineering org if they’ve considered alternative architectures for their main engineering problem is silly at best, overconfident at worst.
  - ben0x539 6y ago
    I think the main point of the comment was asking why they decided against it. At one point the wording mildly suggests the possibility that no one at github has thought about it: > I would love to know if something like this has been considered and why the decided against it. ... but that still sounds more like a grammatical hedge than an actual suggestion that github didn't think it through. imo it's fair to lay out why you're surprised about some decision in the hopes that someone will enlighten you, even if it can be tricky to phrase that without coming off like a "why didn't you just..." comment.
  - brown9-2 6y ago

Ericson2314 6y ago

I always had the impression GitHub was not preemptively investing in the fundamentals like that. So yeah, agree it's bummer but also not surprised. And hey, at least that means a post GitHub FOSS world won't be leaving fundamental improvements behind!

lumost 6y ago

I am not a github employee, but my 2 cents. An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS's can be distributed to arbitrary write velocity given an appropriately distributed block storage solution ( which will provide the k/v API of an object store that you're looking for ). Distributed FS's are conveniently compatible with most POSIX operations rather than requiring bespoke integration. Most object stores are optimized for largish objects and lack the ability to condense records into an individual write (via the block API) or pre-emptively pre-fetch the most likely next set of requested blocks. In the GitHub's case the choice of diverging from GitCLI/FS based storage APIs could lead to long term support issues and an implicit "github" flavor of git rather than improving the core git toolchain. Object Stores are great, but if you need some form of index they get slow and painful really fast.

ddorian43 6y ago
You should be able to split the object store into 2 systems: 1 metadata (think rdbms/nosql/etc) and a blob-data service, keeping large files, think 10KB+. Both systems should be able to be more efficient than the current method. Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc. But somebody has to create it, that's the issue.
- lumost 6y ago
  That's exactly how distributed filesystems are built. Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.
merb 6y ago
https://github.com/juicedata/juicefs https://github.com/juicedata/juicefs has an index implementation and is backed up by object storage.
- 6y ago

hobofan 6y ago

> I'm slightly surprised that GitHub is still basically storing a git repo on a regular filesystem using the Git CLI. Maybe I'm a bit dense, but how did you get that from the article? I'm fairly certain that in other pieces of writing they showed that they are using an object store, and I'm guessing that's what the "file servers" in the article are.

stuhood 6y ago
`git repack` is an operation that is fairly specific to git's default file format: if they were storing objects in any (other) database, it is very unlikely that they would experience blocking repack operations, as that is an area where databases are highly optimized to execute incrementally.

oconnor663 6y ago

I think the GitHub folks have written more than one article about this. I'm not sure I can find the one I'm thinking of, but here's another one: https://github.blog/2016-04-05-introducing-dgit/ https://github.blog/2016-04-05-introducing-dgit/ > Perhaps it’s surprising that GitHub’s repository-storage tier, DGit, is built using the same technologies. Why not a SAN? A distributed file system? Some other magical cloud technology that abstracts away the problem of storing bits durably? The answer is simple: it’s fast and it’s robust.

mvzvm 6y ago

This kind of comment is why every single project needs to be justified with "What problems are you solving?" and "What usecase are you supporting?". Because I could 150% imagine somebody getting excited about this and then: 1) Framing it as such with poor justification "a lot easier to manage" 2) "This would be a huge rewrite of some internals" Becoming a multi-year migration quagmire 3) The dawning realization that you have used a write-heavy architecture in a read-heavy system

saurik 6y ago
I actually implemented this one day--storing all of the git objects into a PostgreSQL database, as a modification to the normal C program copy of git (not some go fork or "libgit2" or whatever people seem to expect you have to use)--and it was really easy: git already is designed to do its object access via a simple internal abstraction, so it was a trivial tap.
- jhatemyjob 6y ago
  Why postgres? Wouldn't SQLite be better?

wmf 6y ago

Vincent Marti's legendary and funny "My Mom Told Me That git Doesn't Scale" presentation explains why GitHub chose filesystems instead of something fancier. https://www.youtube.com/watch?v=Ri8hSZNKzu4 https://www.youtube.com/watch?v=Ri8hSZNKzu4

iudqnolq 6y ago
Thanks, I hadn't seen that before. I really liked the line "People think we're a Ruby shop, but actually we're a UNIX shop. Everything else is an implementation detail".

remram 6y ago

> infinitely scalable writes Scalable receive, but having to send hundred of thousands of objects one-by-one is very inefficient. The fact that there are hundreds of thousands of servers to receive them does not help you. (and same problem when you pull)

gh-throw 6y ago

Not to pile on (as the idea isn't necessarily, immediately crazy): 1) The filesystem is a surprisingly decent datastore, and, 2) Unless your write-volume is unreasonably high for a GH repo, your main problem is making sure your locking (and unlocking) is rock-fucking-solid; Git has built-in locking, but I bet Github has either replaced it or has some serious state-machine stuff going on to manage it outside of Git itself, also and, 3) Reads on files are ultra-cacheable, obviously, and reads on HTML pages are too, and serving a second-or-two-stale read isn't a problem ~100% of the time in GH's use case, so read volume isn't really a problem. [EDIT] source: have designed and developed a git-backed, client-accessible product at a much smaller scale than GH, but enough to see where the problems & strengths could/would be, at greater scale. [EDIT EDIT] oh and with a bunch of clients using a bunch of different repos that effectively never interact except through normal git-push/PR behavior, obviously horizontal scaling is a breeze and a half. All you need is a little routing to figure out which repo is where. Git itself, plus your locking solution (whatever that may be), give you the tools you need to migrate from one server to another if you need to move a repo, probably with an unnoticeable amount of downtime (=locked repo) if you're clever about it.