9 ms·

Muxfs – a mirroring, checksumming, and self-healing filesystem layer for OpenBSD

defrost 4y ago
Nice concept, and having skimmed it's worth noting: > muxfs needs you! > No filesystem can be considered stable without thorough testing and muxfs is no exception. > Even if I had tested muxfs enough to call it stable it still would not be responsible to expect you to simply take my word for it. It is for this reason that I do not intend to release a version 1.0 until there are sufficient citations that I can make to positive, third-party evaluations of muxfs. > This is where you can help. > I need volunteers to test muxfs, provide feedback, and periodically publish test results.
- zasdffaa 4y ago
  These requirements of much testing of inherently multithreaded code with a lot riding on not destroying user data suggests a model checker would be an essential piece, perhaps someone more knowledgeable can opine.
  - sdadams 4y ago
    OpenBSD's FUSE does not implement multithreading.
ikiris 4y ago
i can't find "zfs" mentioned once in this guy's doc so my first question is... why not?
- danielparks 4y ago
  From the page: > I decided it was finally time to build a file server to centralize my files and guard them against bit-rot. Although I would have preferred to use OpenBSD due to its straightforward configuration and sane defaults, I was surprised to find that none of the typical NAS filesystems were supported. OpenBSD does not support ZFS.
  - hestefisk 4y ago
    Theo is, perhaps rightfully so, against importing what is effectively a paravirtualized Solaris kernel into the OpenBSD source code in order to run a file system.
    - nix23 4y ago
      Too sad, because the partitioning of OpenBSD is why i don't use it, with ZFS you could just do a dataset throw x^w,nosuid etc on them and give them a quota, with ffs one can bet that you run out of space (earlier or later), in one of the partitions (Workstation NOT Server).
      - hestefisk 4y ago
        You can use your own partitioning though? One for /, one for swap. Done.
        nix23 4y ago
        Yes you can but you cant set stuff lime nosuid etc on /
      - anthk 4y ago
        I doubt it. Even for ports you can still symlink /usr/ports to $HOME/ports, for Scummvm with --enable-all-engines or Eduke32 (Build/GPLv2 license clash, can't be shared as a binary). /usr/local is not small at all by default.
        nix23 4y ago
        Beltalowda 4y ago
        I think they're trying to say is you can just link stuff to $HOME if some filesystem runs out of space (not an endorsement of that view, just an explanation).
- ciprian_craciun 4y ago
  Because ZFS is not supported on OpenBSD. In fact he does mention in the beginning of the article that he was surprised that none of the NAS related file-systems are not supported by OpenBSD. On the other side, ZFS is an overly complicated behemoth, that wants direct access to the block device. Meanwhile `muxfs` works with any already existing file-system (local or remote) and just provides the checksums. So both serve different use-cases.
  - swinglock 4y ago
    Non ZFS filesystems are overly simplified, ignoring the problems they ought to be solving.
    - arghwhat 4y ago
      ZFS is not the simplest solution to the problems that "ought to be solved", and the implementation can be rather annoying - lacking support for hardware configuration changes, using its own cache system sidestepping the one in the kernel, and generally not fitting in with normal filesystem paradigms. And that's not even addressing that incremental sends - a huge feature - was (is?) broken due to holebirth, making it unreliable. btrfs kinda blew up, but it would be nice to have a good and simple reliable filesystem that actually fits in with the others. ZFS is what we're stuck with till then.
      - swinglock 4y ago
        Some of them are grounded in Linux politics. I have hope in bcachefs but best not rush, we've all seen btrfs.
        arghwhat 4y ago
        ZFS was never any prettier on BSD either, so wouldn't blame Linux in that. But yes, bcachefs is somewhat interesting. Or maybe btrfs manages to clean up their act one day.
- laumars 4y ago
  Same reason he doesn’t mention BtrFS and a bunch of other file systems: Because he’s running OpenBSD which doesn’t support ZFS.
ranger_danger 4y ago
> a filesystem should automatically check and repair data as it is accessed rather than processing the entire filesystem tree upon every check or repair job. Except this is not sufficient. Flash storage for example is especially susceptible to random bitrot of data over time regardless of whether or not it is ever accessed or even powered on. Ever tried to plug in an old USB stick or SD card only to find out it was totally busted or unreadable? Scanning the entire filesystem and re-checksumming everything is therefore completely necessary.
- iforgotpassword 4y ago
  There is the muxfs audit subcommand though that does that. I guess the author was trying to say that it shouldn't be the only way to do it, as that opens the door for silently returning corrupted data in-between runs, so you start pondering how low you should set the interval between audit runs etc. I guess with automatic checks on every access you can feel safe running the audit every other month or so.
ciprian_craciun 4y ago
I have only two issues with the `muxfs` implementation as it stands: (1) (And the largest problem) is that it requires stable inodes in order to tie the checksums with the actual files. This means (and it's already stated in the article) you can't copy / move / overwrite any of the underlying files without losing the checksums. (Basically it also removes the possibility of accessing one of the mirrors via NFS, FUSE, or anything that doesn't have stable inodes.) (2) (Based on my reading of the article) it doesn't seem to hold a "log" or "sequence" to identify which of the two mirrors are ahead or if they are in sync. In case of a disconnect / reconnect you need to manually tell `muxfs` which is the "newer" one (by using a `sync` before being able to mount it). (I haven't tested it though, I'm running Linux, but I'm quite interested because just last week I thought "why doesn't one implement a FUSE file-system to add checksums and thus prevent bitrot". `muxfs` also adds mirroring.)
- adrian_b 4y ago
  I am also using checksums to detect bit-rot, but in order to tie them to the files they are stored in extended attributes of the files. Thus they do not depend on the inode numbers. OpenBSD also supports extended file attributes, so using them should be possible. Using extended attributes on Linux or FreeBSD requires a few precautions, because there are still various copying/movement/archiving CLI commands or GUI applications that ignore the extended attributes and also some file systems that do not implement extended attributes, e.g. tmpfs on Linux (which supports only certain kinds of system extended attributes, not those defined by the users) or all not extremely new versions of NFS (only NFSv4 in Linux 5.9 or newer supports xattr, unlike samba, which has supported them for decades, mapping them correctly between different file systems, e.g. XFS on Linux to UFS on FreeBSD), so copying a file via those file systems would lose silently the extended attributes of the files. The extended file attributes have been introduced in 1989, in HPFS for OS/2 version 1.2, and they have been brought to UNIX in XFS, in 1993. 30 years later, it is annoying to see that there are still some programs which pretend to make file copies or file archives, but which can lose the extended file attributes, without any warnings or errors.
  - sdadams 4y ago
sdadams 4y ago
Author here. A big thank you to you all for your interest in muxfs! I will try to answer all of your questions as best I can.
- j_not_j 4y ago
  The concept sounds good, but you also need people to review the FUSE implementation as well as muxfs. And test, and so on. One question: do you plan to implement "concatenation" of filesystems, so you can build one very large muxfs system (10s of terabytes) ?
  - sdadams 4y ago
    The OpenBSD FUSE implementation is in base so it should already be well audited. I don't plan to add "concatenation" as you have described, however this can, in theory, be approximated by layering muxfs on top of multiple RAID0s.
aborsy 4y ago
Other than lack of support for NAS file systems in OpenBSD, is there a reason not to use ZFS (in favor of another file system providing similar features)?
- sdadams 4y ago
  The muxfs source code is a lot smaller than that of ZFS so if security is a concern to you then you might find muxfs easier to audit than ZFS. It also compiles quickly so could be a good match for a source-based system. This said I never aimed to "beat" ZFS.
  - mst 4y ago
    I appreciate how clear you are about the chosen trade-offs and I am also quite impressed by the extent to which the install instructions include "and now here's how you check that worked" commands - I'm sure somebody reading this will think "well, yes, obviously you should include those" but it's not as common as I might like and your version thereof is notably thorough.
    - sdadams 4y ago
      Thanks! I'll try to remember this for any future blog entries.
gigatexal 4y ago
would porting ZFS have been worthwile? I know the CDDL is likely considered toxic to have in the kernel but even in a FUSE way might be worthwhile here, no?
- deleted 4y ago
  [deleted]
- sdadams 4y ago
  Porting ZFS to OpenBSD has been attempted before. See this blog post from 2013: https://flak.tedunangst.com/post/ZFS-on-OpenBSD https://flak.tedunangst.com/post/ZFS-on-OpenBSD
  - gigatexal 4y ago
    Ahh ok. Makes sense. Seems this was to include it in the kernel though. I wonder if I fully user space solution would have been doable? (Unlikely I guess)
    - sdadams 4y ago
      There is a FUSE implementation of ZFS for Linux but it is still in development. Porting it at this stage could be tricky.
deleted 4y ago
[deleted]
pmarreck 4y ago
Do you use forward error correction? If so, which algorithm?
- sdadams 4y ago
  No error correcting codes currently. Just checksums and redundant copies. Currently the supported checksum algorithms are crc32, md5, and sha1.
tbe 4y ago
Would muxfs be easily portable to other operating systems, or does it rely on non-standard APIs?

Nice concept, and having skimmed it's worth noting: > muxfs needs you! > No filesystem can be considered stable without thorough testing and muxfs is no exception. > Even if I had tested muxfs enough to call it stable it still would not be responsible to expect you to simply take my word for it. It is for this reason that I do not intend to release a version 1.0 until there are sufficient citations that I can make to positive, third-party evaluations of muxfs. > This is where you can help. > I need volunteers to test muxfs, provide feedback, and periodically publish test results.

zasdffaa 4y ago
These requirements of much testing of inherently multithreaded code with a lot riding on not destroying user data suggests a model checker would be an essential piece, perhaps someone more knowledgeable can opine.
- sdadams 4y ago
  OpenBSD's FUSE does not implement multithreading.

ikiris 4y ago

i can't find "zfs" mentioned once in this guy's doc so my first question is... why not?

danielparks 4y ago
From the page: > I decided it was finally time to build a file server to centralize my files and guard them against bit-rot. Although I would have preferred to use OpenBSD due to its straightforward configuration and sane defaults, I was surprised to find that none of the typical NAS filesystems were supported. OpenBSD does not support ZFS.
- hestefisk 4y ago
  Theo is, perhaps rightfully so, against importing what is effectively a paravirtualized Solaris kernel into the OpenBSD source code in order to run a file system.
  - nix23 4y ago
    Too sad, because the partitioning of OpenBSD is why i don't use it, with ZFS you could just do a dataset throw x^w,nosuid etc on them and give them a quota, with ffs one can bet that you run out of space (earlier or later), in one of the partitions (Workstation NOT Server).
    - hestefisk 4y ago
      You can use your own partitioning though? One for /, one for swap. Done.
      - nix23 4y ago
        Yes you can but you cant set stuff lime nosuid etc on /
    - anthk 4y ago
      I doubt it. Even for ports you can still symlink /usr/ports to $HOME/ports, for Scummvm with --enable-all-engines or Eduke32 (Build/GPLv2 license clash, can't be shared as a binary). /usr/local is not small at all by default.
      - nix23 4y ago
        Beltalowda 4y ago
        I think they're trying to say is you can just link stuff to $HOME if some filesystem runs out of space (not an endorsement of that view, just an explanation).
ciprian_craciun 4y ago
Because ZFS is not supported on OpenBSD. In fact he does mention in the beginning of the article that he was surprised that none of the NAS related file-systems are not supported by OpenBSD. On the other side, ZFS is an overly complicated behemoth, that wants direct access to the block device. Meanwhile `muxfs` works with any already existing file-system (local or remote) and just provides the checksums. So both serve different use-cases.
- swinglock 4y ago
  Non ZFS filesystems are overly simplified, ignoring the problems they ought to be solving.
  - arghwhat 4y ago
    ZFS is not the simplest solution to the problems that "ought to be solved", and the implementation can be rather annoying - lacking support for hardware configuration changes, using its own cache system sidestepping the one in the kernel, and generally not fitting in with normal filesystem paradigms. And that's not even addressing that incremental sends - a huge feature - was (is?) broken due to holebirth, making it unreliable. btrfs kinda blew up, but it would be nice to have a good and simple reliable filesystem that actually fits in with the others. ZFS is what we're stuck with till then.
    - swinglock 4y ago
      Some of them are grounded in Linux politics. I have hope in bcachefs but best not rush, we've all seen btrfs.
      - arghwhat 4y ago
        ZFS was never any prettier on BSD either, so wouldn't blame Linux in that. But yes, bcachefs is somewhat interesting. Or maybe btrfs manages to clean up their act one day.
laumars 4y ago
Same reason he doesn’t mention BtrFS and a bunch of other file systems: Because he’s running OpenBSD which doesn’t support ZFS.

ranger_danger 4y ago

> a filesystem should automatically check and repair data as it is accessed rather than processing the entire filesystem tree upon every check or repair job. Except this is not sufficient. Flash storage for example is especially susceptible to random bitrot of data over time regardless of whether or not it is ever accessed or even powered on. Ever tried to plug in an old USB stick or SD card only to find out it was totally busted or unreadable? Scanning the entire filesystem and re-checksumming everything is therefore completely necessary.

iforgotpassword 4y ago
There is the muxfs audit subcommand though that does that. I guess the author was trying to say that it shouldn't be the only way to do it, as that opens the door for silently returning corrupted data in-between runs, so you start pondering how low you should set the interval between audit runs etc. I guess with automatic checks on every access you can feel safe running the audit every other month or so.

ciprian_craciun 4y ago

I have only two issues with the `muxfs` implementation as it stands: (1) (And the largest problem) is that it requires stable inodes in order to tie the checksums with the actual files. This means (and it's already stated in the article) you can't copy / move / overwrite any of the underlying files without losing the checksums. (Basically it also removes the possibility of accessing one of the mirrors via NFS, FUSE, or anything that doesn't have stable inodes.) (2) (Based on my reading of the article) it doesn't seem to hold a "log" or "sequence" to identify which of the two mirrors are ahead or if they are in sync. In case of a disconnect / reconnect you need to manually tell `muxfs` which is the "newer" one (by using a `sync` before being able to mount it). (I haven't tested it though, I'm running Linux, but I'm quite interested because just last week I thought "why doesn't one implement a FUSE file-system to add checksums and thus prevent bitrot". `muxfs` also adds mirroring.)

adrian_b 4y ago
I am also using checksums to detect bit-rot, but in order to tie them to the files they are stored in extended attributes of the files. Thus they do not depend on the inode numbers. OpenBSD also supports extended file attributes, so using them should be possible. Using extended attributes on Linux or FreeBSD requires a few precautions, because there are still various copying/movement/archiving CLI commands or GUI applications that ignore the extended attributes and also some file systems that do not implement extended attributes, e.g. tmpfs on Linux (which supports only certain kinds of system extended attributes, not those defined by the users) or all not extremely new versions of NFS (only NFSv4 in Linux 5.9 or newer supports xattr, unlike samba, which has supported them for decades, mapping them correctly between different file systems, e.g. XFS on Linux to UFS on FreeBSD), so copying a file via those file systems would lose silently the extended attributes of the files. The extended file attributes have been introduced in 1989, in HPFS for OS/2 version 1.2, and they have been brought to UNIX in XFS, in 1993. 30 years later, it is annoying to see that there are still some programs which pretend to make file copies or file archives, but which can lose the extended file attributes, without any warnings or errors.
- sdadams 4y ago

sdadams 4y ago

Author here. A big thank you to you all for your interest in muxfs! I will try to answer all of your questions as best I can.

j_not_j 4y ago
The concept sounds good, but you also need people to review the FUSE implementation as well as muxfs. And test, and so on. One question: do you plan to implement "concatenation" of filesystems, so you can build one very large muxfs system (10s of terabytes) ?
- sdadams 4y ago
  The OpenBSD FUSE implementation is in base so it should already be well audited. I don't plan to add "concatenation" as you have described, however this can, in theory, be approximated by layering muxfs on top of multiple RAID0s.

aborsy 4y ago

Other than lack of support for NAS file systems in OpenBSD, is there a reason not to use ZFS (in favor of another file system providing similar features)?

sdadams 4y ago
The muxfs source code is a lot smaller than that of ZFS so if security is a concern to you then you might find muxfs easier to audit than ZFS. It also compiles quickly so could be a good match for a source-based system. This said I never aimed to "beat" ZFS.
- mst 4y ago
  I appreciate how clear you are about the chosen trade-offs and I am also quite impressed by the extent to which the install instructions include "and now here's how you check that worked" commands - I'm sure somebody reading this will think "well, yes, obviously you should include those" but it's not as common as I might like and your version thereof is notably thorough.
  - sdadams 4y ago
    Thanks! I'll try to remember this for any future blog entries.

gigatexal 4y ago

would porting ZFS have been worthwile? I know the CDDL is likely considered toxic to have in the kernel but even in a FUSE way might be worthwhile here, no?

deleted 4y ago
[deleted]
sdadams 4y ago
Porting ZFS to OpenBSD has been attempted before. See this blog post from 2013: https://flak.tedunangst.com/post/ZFS-on-OpenBSD https://flak.tedunangst.com/post/ZFS-on-OpenBSD
- gigatexal 4y ago
  Ahh ok. Makes sense. Seems this was to include it in the kernel though. I wonder if I fully user space solution would have been doable? (Unlikely I guess)
  - sdadams 4y ago
    There is a FUSE implementation of ZFS for Linux but it is still in development. Porting it at this stage could be tricky.

deleted 4y ago

[deleted]

pmarreck 4y ago

Do you use forward error correction? If so, which algorithm?

sdadams 4y ago
No error correcting codes currently. Just checksums and redundant copies. Currently the supported checksum algorithms are crc32, md5, and sha1.

tbe 4y ago

Would muxfs be easily portable to other operating systems, or does it rely on non-standard APIs?