There's still a lot of room for improvement in distributed filesystems - existing distributed filesystems are nowhere near as capable as they ought to be and it

There's still a lot of room for improvement in distributed filesystems - existing distributed filesystems are nowhere near as capable as they ought to be and it's going to require new more or less clean sheet architecture(s) before we get there. There's a lot of hard problems that still haven't had really adequate solutions implemented in production code. Hell, even network filesystems (non distributed) aren't where they should be. To illustrate: there is no fundamental reason why a distributed filesystem shouldn't, in practice, perform more or less equivalently to a network, non distributed filesystem. That's not the case today; existing distributed filesystems have slowpaths - generally, things involving metadata and anything that requires coordinating servers - where they're drastically slower than say existing NFS or 9p implementations, and these aren't obscure corner cases, these are things real applications hit. Reasoning from first principles, we know that: - It's not going to be possible to create a distributed filesystem that doesn't have these slowpaths, it's probably always going to be possible to create an evil/pathalogical workload that requires lots of cross server synchronization, but: - Most real world applications don't behave so badly. Imagine you just want a single filesystem that has a crapton of home directories, and you want to throw it all on a distributed filesystem instead of giant Netapp filers or whatever. We should be able to do that, and have it perform just as well (to with epsilon) as using NFS and explicit sharding. We can't really do that today. In my opinion, we don't really have an architecture implemented today that's capable of solving these problems - that said, I'm not familiar with Ceph's architecture so I don't know how it tackles sharding. I have my own ideas about how it could be done (for awhile, I was working on something on top of the bcache/bcachefs codebase; I actually implemented working, fast distributed block storage, but not filesystem, and it didn't have nearly everything implemented that it would have needed to scale out horizontally very far). But I never got far enough into the filesystem aspect to really prove anything. For network (non distributed) filesystems - again, those aren't as good as they ought to be either. But the problem there is much simpler - nobody's built anything to my knowledge that would let you really do cache coherency correctly: you can't do it by just using an existing local filesystem implementation and serving it over a new protocol, because you can't do cache invalidation correctly without storing fine grained versioning information that local filesystems don't need, and nobody's wanted to try grafting that onto existing local filesystems. A lot of the story of NFS is hacks to get around that problem.