2 ms·

I am not a github employee, but my 2 cents. An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS'

by lumost 6y ago

I am not a github employee, but my 2 cents. An object store lacks an index which your typical FS will provide with a relatively high degree of efficiency. FS's can be distributed to arbitrary write velocity given an appropriately distributed block storage solution ( which will provide the k/v API of an object store that you're looking for ). Distributed FS's are conveniently compatible with most POSIX operations rather than requiring bespoke integration. Most object stores are optimized for largish objects and lack the ability to condense records into an individual write (via the block API) or pre-emptively pre-fetch the most likely next set of requested blocks. In the GitHub's case the choice of diverging from GitCLI/FS based storage APIs could lead to long term support issues and an implicit "github" flavor of git rather than improving the core git toolchain. Object Stores are great, but if you need some form of index they get slow and painful really fast.

ddorian43 6y ago
You should be able to split the object store into 2 systems: 1 metadata (think rdbms/nosql/etc) and a blob-data service, keeping large files, think 10KB+. Both systems should be able to be more efficient than the current method. Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc. But somebody has to create it, that's the issue.
- lumost 6y ago
  That's exactly how distributed filesystems are built. Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.
merb 6y ago
https://github.com/juicedata/juicefs https://github.com/juicedata/juicefs has an index implementation and is backed up by object storage.
- lumost 6y ago
  aye - as it says it provides an FS. The second diagram shows how they chop up files into blocks then store them in the backing object (k/v) storage. A filesystem really is just an index over k/v storage (blocks). You can find similar diagrams for the implementation of EXT/XFS/GFS/HDFS etc. Like databases there are many tradeoffs and nuances in terms of how these concepts are implemented.

by lumost 6y ago

ddorian43 6y ago

You should be able to split the object store into 2 systems: 1 metadata (think rdbms/nosql/etc) and a blob-data service, keeping large files, think 10KB+. Both systems should be able to be more efficient than the current method. Example: you can add erasure coding to the blob-data service for better efficiency. You can add fancy indexing to your metadata store. etc etc. But somebody has to create it, that's the issue.

lumost 6y ago
That's exactly how distributed filesystems are built. Systems such as HDFS use the NameNode for this task, but depending on the exact characteristics of the fileSystem a multi-master setup is often used. I know of at least one NFS implementation which uses postgres as its metadata layer.

merb 6y ago

https://github.com/juicedata/juicefs https://github.com/juicedata/juicefs has an index implementation and is backed up by object storage.

lumost 6y ago
aye - as it says it provides an FS. The second diagram shows how they chop up files into blocks then store them in the backing object (k/v) storage. A filesystem really is just an index over k/v storage (blocks). You can find similar diagrams for the implementation of EXT/XFS/GFS/HDFS etc. Like databases there are many tradeoffs and nuances in terms of how these concepts are implemented.