4 ms·

ok i dont get this ... shouldn't server availability problems be solved using traditional server availability technologies, like cloud technologies why mess wi

by systems 11y ago

ok i dont get this ... shouldn't server availability problems be solved using traditional server availability technologies, like cloud technologies why mess with git

shadowmint 11y ago
Like what? I'm going to tentatively suggest this is one of those 'hard' problems that throwing buzz words like 'cloud technologies' at doesn't solve. What replication tech would you imagine solves this issue of distributing hundreds of thousands of constantly updated repositories?
- justinsb 11y ago
  It's actually a relatively easy problem (compared to say a full POSIX filesystem) - I mentioned elsewhere in these comments a blog post where I implemented a fairly good solution. The objects are essentially immutable and content-addressable so you can get away with very relaxed semantics here. You need a reliable mapping of names to SHAs, but this is also comparatively easy (a key-value store). For example, you can easily satisfy this with S3 and DynamoDB - I think the latest version of the cloudata project I was blogging about actually does that now.
  - summner 11y ago
    They've addressed that in the post. To make git work comfortably especially on bigger repositories you need to have fast local access to all blobs. IIRC with something like libgit2 its relatively easy to implement what you describe but to make that perform while doing git log or diffs is completely different story.
    - justinsb 11y ago
      It isn't clear that they wouldn't get that same performance by caching the blobs locally. In my experience, you would, with much higher reliability and scalability.
wmf 11y ago
There's a classic talk by vmg that explains GitHub's philosophy really well: http://devslovebacon.com/conferences/bacon-2013/talks/my-mom-told-me-that-git-doesnt-scale http://devslovebacon.com/conferences/bacon-2013/talks/my-mom... (it gets webscale around the 15 minute mark)

by systems 11y ago

ok i dont get this ... shouldn't server availability problems be solved using traditional server availability technologies, like cloud technologies why mess with git

shadowmint 11y ago

Like what? I'm going to tentatively suggest this is one of those 'hard' problems that throwing buzz words like 'cloud technologies' at doesn't solve. What replication tech would you imagine solves this issue of distributing hundreds of thousands of constantly updated repositories?

justinsb 11y ago
It's actually a relatively easy problem (compared to say a full POSIX filesystem) - I mentioned elsewhere in these comments a blog post where I implemented a fairly good solution. The objects are essentially immutable and content-addressable so you can get away with very relaxed semantics here. You need a reliable mapping of names to SHAs, but this is also comparatively easy (a key-value store). For example, you can easily satisfy this with S3 and DynamoDB - I think the latest version of the cloudata project I was blogging about actually does that now.
- summner 11y ago
  They've addressed that in the post. To make git work comfortably especially on bigger repositories you need to have fast local access to all blobs. IIRC with something like libgit2 its relatively easy to implement what you describe but to make that perform while doing git log or diffs is completely different story.
  - justinsb 11y ago
    It isn't clear that they wouldn't get that same performance by caching the blobs locally. In my experience, you would, with much higher reliability and scalability.

wmf 11y ago

There's a classic talk by vmg that explains GitHub's philosophy really well: http://devslovebacon.com/conferences/bacon-2013/talks/my-mom-told-me-that-git-doesnt-scale http://devslovebacon.com/conferences/bacon-2013/talks/my-mom... (it gets webscale around the 15 minute mark)