5 ms·
Buried towards the bottom is the interesting decision to dump CephFS and go with NFS instead.
by frabbit 10y ago
Buried towards the bottom is the interesting decision to dump CephFS and go with NFS instead.
- chillydawg 10y agoFundamentally not much different, though. N shards holding the data, you have to know which shard is which and maintain connections to it. Split brains could happen if that sharding is dynamic etc.
- tra3 10y agoI find that interesting as well. Furthermore, looking into deploying containers with cheap cloud hosters , I was curious how to share data across various nodes by looking at how Google and Amazon do it. Turns out it's still NFS. I've never looked into networked FS before, but I thought they'd move past NFS to something more advanced...and newer I suppose.
- Rezo 10y agoI wonder if Amazon's new managed EFS service would make sense? It's exposed as a NFS mount to the OS. The claims are: - Up to thousands of Amazon EC2 instances, from multiple AZs, can connect concurrently to a file system. - Data is stored redundantly across multiple AZs. - Low, consistent latency. - Multiple GBs per second.
- sytse 10y agoEFS is pretty great. But for our use case there are a few drawbacks: 1. It doesn't scale to the size we need for all repositories, so we still need sharding. 2. It is expensive, $300/TB-month 3. You're still sending all traffic over the network, we prefer the latency of local storage we can achieve by developing Gitaly https://gitlab.com/gitlab-org/gitaly https://gitlab.com/gitlab-org/gitaly
- jamesmiller5 10y agoWhy develop Gitaly instead of using consensus (etcd, zookeeper, etc.) for writes to fast storage like ssds and the normal git binary? As far as I can tell it's adding a lot of complexity by caching high level RPC calls but still doesn't really address multi-master read after write consistency or coordination.