4 ms·

I was wondering why so many distributed file systems - all claiming high performance / availability / scalability - are being implemented from scratch, when mat

I was wondering why so many distributed file systems - all claiming high performance / availability / scalability - are being implemented from scratch, when mature ones like Gluster and Ceph have been around from years. Why start new ones instead of improving existing ones? My guess is that these new infra software like SeaweedFS and LeoFS are created mainly to implement them in new languages like Go, Rust and Erlang. I'm not saying it's a bad thing, just that it's one possible reason so many alternatives are being implemented from scratch.

jdcarter 10y ago
I'm reading between the lines here, and I'd love for Chris Lu (the SeaweedFS author) to chime in here and correct me, but it seems like Chris read Facebook's Haystack paper and decided to implement it himself in Go. Kind of how Riak started as "let's take the DynamoDB paper and implement it in Erlang." The README mentions that Seaweed is much easier to set up than Ceph. The more apt comparison would be the RADOS component of Ceph--both Seaweed and RADOS are key/value oriented. I've now set up both, and for sure Seaweed is easier to get going. I have a fair bit of experience with RADOS (all quite positive) but I really do appreciate the ease of use Seaweed presents. I've only been using it for an hour or so, just doing benchmarks and walking through fault scenarios, and so far Seaweed looks quite good! FWIW, I don't think this should be called a filesystem at all. Seaweed isn't a filesystem which forsakes some aspects of POSIX, it's nothing like a filesystem. The more proper term would be "distributed object store."
- jusob 10y ago
  You are correct, it is not really a file system. It is used with the HTTP API (upload, download, etc.).
  - deleted 10y ago
    [deleted]
- tomphoolery 10y ago
  > but it seems like Chris read Facebook's Haystack paper and decided to implement it himself in Go That definitely sounds like the initial impetus for this project, and a perfectly good reason to make "yet another" file system. Additionally, I can understand how taking POSIX compatibility out of the requirements might make writing and maintaining this project a bit more fun.
- SEJeff 10y ago
  I'd love to see someone run aphyr's Jepsen tool on this and see if it lives up to what it aspires to be. Kyle has a real way of destroying databases and really any distributed datastore: https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen
rphlx 10y ago
Not to trash talk ceph, but it has had some substantial performance & resiliency issues which are only beginning to be addressed in a production-ready way in 2017 (via bluestore, multi-MDS cephfs, local SSD ordered writeback caching for VMs, etc).
- jdcarter 10y ago
  I've been following Ceph (testing it but not using it in production) because of my interest in using RADOS for a project. BlueStore, hot/warm tiering, erasure coded pools are all pretty darn nice features. Are you aware of ongoing issues at the RADOS level? In my past explorations of Ceph's higher level pieces, e.g. RGW and CephFS, those appeared much less-baked than RADOS. That experience was a couple years old, however, and they may have fixed all that since.
  - rphlx 10y ago
    My main concern has been FileStore i.e. running an obj store on top of btrfs/xfs/zfs which even an armchair storage hobbyist (like me) can see, from first principles, is a truly awful idea (journal-on-journal, etc). Bluestore will fix that at long last.
chrislusf 10y ago
Not a simple re-write. My original plan is to break normal file system into composable pieces, and make the file system more scalable. The first piece is the key~blob storage, which consists of master and volume servers. The second piece is the filer server. The first piece should be self contained and agnostic of the layer above, yet still easy-to-use in normal cases where only key~blob service is needed.
jamesblonde 10y ago
Not everybody takes that approach. We developed HopsFS by re-implementing the metadata layer for HDFS as a distributed scale-out layer (paper being presented in Usenix FAST next month - https://www.usenix.org/conference/fast17/technical-sessions https://www.usenix.org/conference/fast17/technical-sessions). As a researcher, this was not a good idea, as it has taken us many years. Time-to-market for our paper was extremely slow, but the upshot is that we have a production system. The big risk for us was that our work would not be seen as relevant when we were finished, and it has been a high risk research angle to take.
- chrislusf 10y ago
  Good to know this! The metadata layer with NewSQL storage seems exactly what SeaweedFS is already doing.
- lovelearning 10y ago
  Very interesting, I'll try out Hops. Hadn't come across it before - perhaps you can post a "Show HN" for others like me. And best wishes for your paper presentation!
c22 10y ago
Is Erlang really a new language? I recall first looking at it in the '90s.
- lovelearning 10y ago
  You are right, but it seems to me Erlang became _popular_ somewhat recently when functional programming started getting attention. CouchDB was the first major project in Erlang that I'd come across, and that was around 6-7 years ago.
koverstreet 10y ago
There's still a lot of room for improvement in distributed filesystems - existing distributed filesystems are nowhere near as capable as they ought to be and it's going to require new more or less clean sheet architecture(s) before we get there. There's a lot of hard problems that still haven't had really adequate solutions implemented in production code. Hell, even network filesystems (non distributed) aren't where they should be. To illustrate: there is no fundamental reason why a distributed filesystem shouldn't, in practice, perform more or less equivalently to a network, non distributed filesystem. That's not the case today; existing distributed filesystems have slowpaths - generally, things involving metadata and anything that requires coordinating servers - where they're drastically slower than say existing NFS or 9p implementations, and these aren't obscure corner cases, these are things real applications hit. Reasoning from first principles, we know that: - It's not going to be possible to create a distributed filesystem that doesn't have these slowpaths, it's probably always going to be possible to create an evil/pathalogical workload that requires lots of cross server synchronization, but: - Most real world applications don't behave so badly. Imagine you just want a single filesystem that has a crapton of home directories, and you want to throw it all on a distributed filesystem instead of giant Netapp filers or whatever. We should be able to do that, and have it perform just as well (to with epsilon) as using NFS and explicit sharding. We can't really do that today. In my opinion, we don't really have an architecture implemented today that's capable of solving these problems - that said, I'm not familiar with Ceph's architecture so I don't know how it tackles sharding. I have my own ideas about how it could be done (for awhile, I was working on something on top of the bcache/bcachefs codebase; I actually implemented working, fast distributed block storage, but not filesystem, and it didn't have nearly everything implemented that it would have needed to scale out horizontally very far). But I never got far enough into the filesystem aspect to really prove anything. For network (non distributed) filesystems - again, those aren't as good as they ought to be either. But the problem there is much simpler - nobody's built anything to my knowledge that would let you really do cache coherency correctly: you can't do it by just using an existing local filesystem implementation and serving it over a new protocol, because you can't do cache invalidation correctly without storing fine grained versioning information that local filesystems don't need, and nobody's wanted to try grafting that onto existing local filesystems. A lot of the story of NFS is hacks to get around that problem.

jdcarter 10y ago

I'm reading between the lines here, and I'd love for Chris Lu (the SeaweedFS author) to chime in here and correct me, but it seems like Chris read Facebook's Haystack paper and decided to implement it himself in Go. Kind of how Riak started as "let's take the DynamoDB paper and implement it in Erlang." The README mentions that Seaweed is much easier to set up than Ceph. The more apt comparison would be the RADOS component of Ceph--both Seaweed and RADOS are key/value oriented. I've now set up both, and for sure Seaweed is easier to get going. I have a fair bit of experience with RADOS (all quite positive) but I really do appreciate the ease of use Seaweed presents. I've only been using it for an hour or so, just doing benchmarks and walking through fault scenarios, and so far Seaweed looks quite good! FWIW, I don't think this should be called a filesystem at all. Seaweed isn't a filesystem which forsakes some aspects of POSIX, it's nothing like a filesystem. The more proper term would be "distributed object store."

jusob 10y ago
You are correct, it is not really a file system. It is used with the HTTP API (upload, download, etc.).
- deleted 10y ago
  [deleted]
tomphoolery 10y ago
> but it seems like Chris read Facebook's Haystack paper and decided to implement it himself in Go That definitely sounds like the initial impetus for this project, and a perfectly good reason to make "yet another" file system. Additionally, I can understand how taking POSIX compatibility out of the requirements might make writing and maintaining this project a bit more fun.
SEJeff 10y ago
I'd love to see someone run aphyr's Jepsen tool on this and see if it lives up to what it aspires to be. Kyle has a real way of destroying databases and really any distributed datastore: https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen

rphlx 10y ago

Not to trash talk ceph, but it has had some substantial performance & resiliency issues which are only beginning to be addressed in a production-ready way in 2017 (via bluestore, multi-MDS cephfs, local SSD ordered writeback caching for VMs, etc).

jdcarter 10y ago
I've been following Ceph (testing it but not using it in production) because of my interest in using RADOS for a project. BlueStore, hot/warm tiering, erasure coded pools are all pretty darn nice features. Are you aware of ongoing issues at the RADOS level? In my past explorations of Ceph's higher level pieces, e.g. RGW and CephFS, those appeared much less-baked than RADOS. That experience was a couple years old, however, and they may have fixed all that since.
- rphlx 10y ago
  My main concern has been FileStore i.e. running an obj store on top of btrfs/xfs/zfs which even an armchair storage hobbyist (like me) can see, from first principles, is a truly awful idea (journal-on-journal, etc). Bluestore will fix that at long last.

chrislusf 10y ago

Not a simple re-write. My original plan is to break normal file system into composable pieces, and make the file system more scalable. The first piece is the key~blob storage, which consists of master and volume servers. The second piece is the filer server. The first piece should be self contained and agnostic of the layer above, yet still easy-to-use in normal cases where only key~blob service is needed.

jamesblonde 10y ago

Not everybody takes that approach. We developed HopsFS by re-implementing the metadata layer for HDFS as a distributed scale-out layer (paper being presented in Usenix FAST next month - https://www.usenix.org/conference/fast17/technical-sessions https://www.usenix.org/conference/fast17/technical-sessions). As a researcher, this was not a good idea, as it has taken us many years. Time-to-market for our paper was extremely slow, but the upshot is that we have a production system. The big risk for us was that our work would not be seen as relevant when we were finished, and it has been a high risk research angle to take.

chrislusf 10y ago
Good to know this! The metadata layer with NewSQL storage seems exactly what SeaweedFS is already doing.
lovelearning 10y ago
Very interesting, I'll try out Hops. Hadn't come across it before - perhaps you can post a "Show HN" for others like me. And best wishes for your paper presentation!

c22 10y ago

Is Erlang really a new language? I recall first looking at it in the '90s.

lovelearning 10y ago
You are right, but it seems to me Erlang became _popular_ somewhat recently when functional programming started getting attention. CouchDB was the first major project in Erlang that I'd come across, and that was around 6-7 years ago.

koverstreet 10y ago

There's still a lot of room for improvement in distributed filesystems - existing distributed filesystems are nowhere near as capable as they ought to be and it's going to require new more or less clean sheet architecture(s) before we get there. There's a lot of hard problems that still haven't had really adequate solutions implemented in production code. Hell, even network filesystems (non distributed) aren't where they should be. To illustrate: there is no fundamental reason why a distributed filesystem shouldn't, in practice, perform more or less equivalently to a network, non distributed filesystem. That's not the case today; existing distributed filesystems have slowpaths - generally, things involving metadata and anything that requires coordinating servers - where they're drastically slower than say existing NFS or 9p implementations, and these aren't obscure corner cases, these are things real applications hit. Reasoning from first principles, we know that: - It's not going to be possible to create a distributed filesystem that doesn't have these slowpaths, it's probably always going to be possible to create an evil/pathalogical workload that requires lots of cross server synchronization, but: - Most real world applications don't behave so badly. Imagine you just want a single filesystem that has a crapton of home directories, and you want to throw it all on a distributed filesystem instead of giant Netapp filers or whatever. We should be able to do that, and have it perform just as well (to with epsilon) as using NFS and explicit sharding. We can't really do that today. In my opinion, we don't really have an architecture implemented today that's capable of solving these problems - that said, I'm not familiar with Ceph's architecture so I don't know how it tackles sharding. I have my own ideas about how it could be done (for awhile, I was working on something on top of the bcache/bcachefs codebase; I actually implemented working, fast distributed block storage, but not filesystem, and it didn't have nearly everything implemented that it would have needed to scale out horizontally very far). But I never got far enough into the filesystem aspect to really prove anything. For network (non distributed) filesystems - again, those aren't as good as they ought to be either. But the problem there is much simpler - nobody's built anything to my knowledge that would let you really do cache coherency correctly: you can't do it by just using an existing local filesystem implementation and serving it over a new protocol, because you can't do cache invalidation correctly without storing fine grained versioning information that local filesystems don't need, and nobody's wanted to try grafting that onto existing local filesystems. A lot of the story of NFS is hacks to get around that problem.