4 ms·

Yes filesystems are another big important part when it comes to the topic of data consistency. I'm not aware of a project similar to Jepsen for filesystems but

by eis 6y ago

Yes filesystems are another big important part when it comes to the topic of data consistency. I'm not aware of a project similar to Jepsen for filesystems but there have been research papers like the one presented in this page that explore the guarantees of popular local filesystems: https://danluu.com/file-consistency/ https://danluu.com/file-consistency/ Of course distributed filesystems are even harder but we can see that even writing to a local file is surprisingly complex and in many (most?) apps probably not really guaranteeing proper data consistency. Imho the whole Posix filesystem API is flawed to begin with and it would be great if a modern replacement would emerge.

thekozmo 6y ago
We did something in the neighborhood, check it out: https://www.scylladb.com/2016/02/16/fault-injection-filesystem-software-testing/ https://www.scylladb.com/2016/02/16/fault-injection-filesyst...
redis_mlc 6y ago
> I'm not aware of a project similar to Jepsen for filesystems Not as elaborate as Jepsen, but there has been some work: - Kirk McKusick's papers and work on BSD fs log semantics https://www.researchgate.net/scientific-contributions/Marshall-Kirk-McKusick-70507097 https://www.researchgate.net/scientific-contributions/Marsha... I believe the term to look for is "soft updates." - the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version. You can Bing versions of that by using quotes "fstest.c". - the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.) - Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines. - Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production. SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out. https://en.wikipedia.org/wiki/Dancing_tree https://en.wikipedia.org/wiki/Dancing_tree Source: DBA and storage engineer.

by eis 6y ago

thekozmo 6y ago

We did something in the neighborhood, check it out: https://www.scylladb.com/2016/02/16/fault-injection-filesystem-software-testing/ https://www.scylladb.com/2016/02/16/fault-injection-filesyst...

redis_mlc 6y ago

> I'm not aware of a project similar to Jepsen for filesystems Not as elaborate as Jepsen, but there has been some work: - Kirk McKusick's papers and work on BSD fs log semantics https://www.researchgate.net/scientific-contributions/Marshall-Kirk-McKusick-70507097 https://www.researchgate.net/scientific-contributions/Marsha... I believe the term to look for is "soft updates." - the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version. You can Bing versions of that by using quotes "fstest.c". - the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.) - Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines. - Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production. SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out. https://en.wikipedia.org/wiki/Dancing_tree https://en.wikipedia.org/wiki/Dancing_tree Source: DBA and storage engineer.