4 ms·
I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system serve
by derefr 13y ago
I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem, especially if you also allowed composite file objects, and perhaps per-object concurrent BitTorrent retrieval.
- rpedela 13y agoIsn't that what git-annex is for? https://git-annex.branchable.com/ https://git-annex.branchable.com/
- platz 13y agoAlso some interesting related projects on https://git-annex.branchable.com/not/ https://git-annex.branchable.com/not/
- derefr 13y agogit-annex is an ugly hack to, basically, reimplement git-with-this-feature on top of git. What I'm suggesting would be to reimplement git itself on top of distributed object storage -- effectively, to split git into two tools: 1. one that commits treeish diffs into objects, and then stuffs them into arbitrary content-addressable storage; 2. and one that manipulates (local or remote) mutable tables of refs to content-hash-URNs -- or "git repositories" for short. The key component would be a facility (either a library or an OS service) for resolving content-hash-URNs into file-descriptors in some content-addressable storage system. The critical point is that this would not be a git component -- the committing tool wouldn't be in charge of where its objects go, and the repository tool wouldn't be in charge of resolving their URNs. Instead, both would just be relying on the host to decide how to get objects from/put objects into storage, and it's the host that would have a distribution strategy set up for all of its content-addressable objects, not just the ones that happen to be attached to git.
- michaelmior 13y agoYou would end up with pretty huge latency for a lot of common operations in that case. Every checkout, diff, and status for example.
- zobzu 13y agoyeah it sounds like svn all over again
- derefr 13y agoThat depends on how replicated/distributed the object storage is. In a DHT scenario, you might already be storing some of the objects yourself before you've even requested them. The remote server can obviously be running an object-storage node itself, and so can your own local git repo -- the point is that you can have objects in more places than just your machine and the repo it's pulling from.
- espeed 13y agoYou can use JGit (https://github.com/eclipse/jgit https://github.com/eclipse/jgit) to store to S3. See http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a-private-git-repository/ http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...
- beliu 13y agoSee also: https://camlistore.org/ https://camlistore.org/ (unlike Dat, doesn't target datasets, but more generic user data, like photos and files; more Dropbox than npm). It has a pretty cool blob model, exposes a FUSE interface, has a slick photo-browsing UI, and integrates with S3 and (I think) Glacier.
- sciurus 13y agoFor a recent look at camlistore see http://lwn.net/SubscriberLink/595746/2adfdecdb0de2577/ http://lwn.net/SubscriberLink/595746/2adfdecdb0de2577/ (and if you like that article, consider subscribing to LWN!)
- addisaden 13y agojust plain git with also large datafiles triggers my mind a lot. not databases ... just all kinds of data. (like large pdfs) ..
- TillE 13y agoYou could probably modify Mercurial's largefiles extension to use S3, etc as a central store.
- beagle3 13y agoLibgit2 does this internally, although it doesn't expose the underlying store (so you wouldn't know to use S3 even if you could get objects from there)