5 ms·
> Bonus question) Why are we thinking about all of this. You should have just used s3? NFS and Ceph both support the standard POSIX file access model, notably
by geofft 10y ago
> Bonus question) Why are we thinking about all of this. You should have just used s3?
NFS and Ceph both support the standard POSIX file access model, notably including random-access reads and writes on a file without replacing the entire file and consistency models that let you reason about directories and their contents usefully. If your goal is to run the reference implementation of git, or something that is compatible with it, on the filesystem, you absolutely need something that supports this.
You could write something new that speaks the git protocol but is designed to be backed by an eventually-consistent object store instead of a POSIX-compliant file store ... but that seems like an equally big challenge, honestly.
- kyrra 10y agoFun aside. Google wrote a backend to Mercurial so it could be stored on bigtable. They did a talk about it 7 years ago. http://m.youtube.com/watch?v=ri796Hx8las http://m.youtube.com/watch?v=ri796Hx8las
- geofft 10y agoAnd to SVN! https://lwn.net/Articles/194667/ https://lwn.net/Articles/194667/ It's not a bad decision; getting POSIX semantics to play well with modern expectations of a highly-consistent and highly-available distributed system is very difficult. But it's a work-intensive decision, and figuring out how to deploy NFS or Ceph at scale might be less work. Especially if Google's SVN/hg developers can work directly with the Bigtable developers on features, and GitLab's developers can't work directly with the S3 team (but can own their own Ceph or NFS stack).
- zaphar 10y agoTheir Git hosting was also running on that same backend (I was on the team managing those products). But to a previous posters point That is a lot of work both to implement as well as to maintain. If you can just use a distributed filesystem you might save yourself some trouble.
- Rapzid 10y agoOthers have done this with libgit2 storage backends.
- dragonne 10y agoAnyone in particular, or anything open source? I could use this for a work project, but the closest I've been able to find is an old project which backed JGit to Cassandra and the Dulwich Swift backend[1]. [1]: https://www.dulwich.io/apidocs/dulwich.contrib.swift.html https://www.dulwich.io/apidocs/dulwich.contrib.swift.html
- sytse 10y agoWe looked at this but it didn't have either the features or the speed we needed. I do think it is very promising.
- Maarten88 10y agoYes, write a Git backend using a distributed actor system, like Akka, Orleans or Service Fabric, backed by Table Storage, Bigtable or some other distributed hashtable store. Git maps naturally to this structure, using git hashes as table index. You'll have a really cheap and very scalable solution that uses low level storage mechanisms yet it will also be really performant because when loaded everything will be served from memory.
- sytse 10y agoFor sure, and people are trying to do this on the basis of libgit2. But so far nobody has been able to make it work for all of the cases. Even GitHub is using filesystems as far as I know.
- markelliot 10y agoOne of the teams I work closely with used JGit's extensibility to run git on top of a KVS: https://medium.com/@palantir/stemma-distributed-git-server-70afbca0fc29 https://medium.com/@palantir/stemma-distributed-git-server-7... Notably this doesn't require a distributed filesystem.
- sytse 10y agoThat looks really interesting, thanks for posting. I added it to our issue https://gitlab.com/gitlab-com/infrastructure/issues/727#note_18253855 https://gitlab.com/gitlab-com/infrastructure/issues/727#note...
- nixgeek 10y agoRelevant: http://githubengineering.com/introducing-dgit/ http://githubengineering.com/introducing-dgit/
- ethomson 10y agoFor Visual Studio Online and TFS, Microsoft is using libgit2 and some custom ASP.NET code for the upload-pack / receive-pack endpoints, stuffing the data into SQL Azure. No filesystem, no `git` executable.