5 ms·

I have only two issues with the `muxfs` implementation as it stands: (1) (And the largest problem) is that it requires stable inodes in order to tie the checks

I have only two issues with the `muxfs` implementation as it stands: (1) (And the largest problem) is that it requires stable inodes in order to tie the checksums with the actual files. This means (and it's already stated in the article) you can't copy / move / overwrite any of the underlying files without losing the checksums. (Basically it also removes the possibility of accessing one of the mirrors via NFS, FUSE, or anything that doesn't have stable inodes.) (2) (Based on my reading of the article) it doesn't seem to hold a "log" or "sequence" to identify which of the two mirrors are ahead or if they are in sync. In case of a disconnect / reconnect you need to manually tell `muxfs` which is the "newer" one (by using a `sync` before being able to mount it). (I haven't tested it though, I'm running Linux, but I'm quite interested because just last week I thought "why doesn't one implement a FUSE file-system to add checksums and thus prevent bitrot". `muxfs` also adds mirroring.)

adrian_b 4y ago
I am also using checksums to detect bit-rot, but in order to tie them to the files they are stored in extended attributes of the files. Thus they do not depend on the inode numbers. OpenBSD also supports extended file attributes, so using them should be possible. Using extended attributes on Linux or FreeBSD requires a few precautions, because there are still various copying/movement/archiving CLI commands or GUI applications that ignore the extended attributes and also some file systems that do not implement extended attributes, e.g. tmpfs on Linux (which supports only certain kinds of system extended attributes, not those defined by the users) or all not extremely new versions of NFS (only NFSv4 in Linux 5.9 or newer supports xattr, unlike samba, which has supported them for decades, mapping them correctly between different file systems, e.g. XFS on Linux to UFS on FreeBSD), so copying a file via those file systems would lose silently the extended attributes of the files. The extended file attributes have been introduced in 1989, in HPFS for OS/2 version 1.2, and they have been brought to UNIX in XFS, in 1993. 30 years later, it is annoying to see that there are still some programs which pretend to make file copies or file archives, but which can lose the extended file attributes, without any warnings or errors.
- sdadams 4y ago
  OpenBSD removed support for extended attributes.
  - minusf 4y ago
    not a 1:1 equivalent of the "classic" extended attr but there are special flags on files. https://man.openbsd.org/chflags https://man.openbsd.org/chflags
  - adrian_b 4y ago
    I suppose that this has been done to ensure less work for the OpenBSD maintainers, but I do not consider it as a wise decision. Extended file attributes can be used to implement a large number of useful things, many of them enhancing security, e.g. access-control lists. However, IIRC OpenBSD has chosen to also not implement ACLs. In any case regardless how useful or not useful extended file attributes are considered to be, deciding to not implement them in the main file system used by an operating system has the immediate consequence of disqualifying this operating system for the use on a file server (a.k.a. NAS), which is an application domain where the *BSD operating systems have been traditionally very good. The reason is that whenever such a NAS would have Windows, Linux or FreeBSD clients, transiting any file through that NAS would potentially lose data. In general, in the documentation of any file system, the lack of support for features like extended attributes or access-control lists must be displayed very prominently, to warn any potential users about the risk of data loss during copy operations (because the file copy commands are usually stupid enough to not inform the users whenever they are stripping file metadata, so such a loss may be discovered only when it is too late).
    - sdadams 4y ago
      In another project of mine I considered using extended attributes to tag files into categories. With extended attributes not being universally supported and easily overlooked the conclusion I came to was that I should store that information in a database file. I didn't want to one day lose my (manually assigned) tags to an erroneous move command. Anything non-native to OpenBSD can still be stored in muxfs as an archive file. You would need to ensure to pass the right arguments to the archiver to preserve the attributes in this case.
sdadams 4y ago
(1) I believe that stable inode numbers are possible with FUSE but they must be implemented by the FUSE driver. Mirroring over a network is not the intended use case. The mirrors exist to provide data redundancy for use by the muxfs driver. If you need to copy the data to a new location there is the muxfs sync command. (2) muxfs uses sequence numbers to count the write operations performed on each mirror. Upon failure to mount due to the mirrors being out of sync a report is printed comparing the first mirror with the first non-matching mirror, and this includes their sequence numbers.

adrian_b 4y ago

I am also using checksums to detect bit-rot, but in order to tie them to the files they are stored in extended attributes of the files. Thus they do not depend on the inode numbers. OpenBSD also supports extended file attributes, so using them should be possible. Using extended attributes on Linux or FreeBSD requires a few precautions, because there are still various copying/movement/archiving CLI commands or GUI applications that ignore the extended attributes and also some file systems that do not implement extended attributes, e.g. tmpfs on Linux (which supports only certain kinds of system extended attributes, not those defined by the users) or all not extremely new versions of NFS (only NFSv4 in Linux 5.9 or newer supports xattr, unlike samba, which has supported them for decades, mapping them correctly between different file systems, e.g. XFS on Linux to UFS on FreeBSD), so copying a file via those file systems would lose silently the extended attributes of the files. The extended file attributes have been introduced in 1989, in HPFS for OS/2 version 1.2, and they have been brought to UNIX in XFS, in 1993. 30 years later, it is annoying to see that there are still some programs which pretend to make file copies or file archives, but which can lose the extended file attributes, without any warnings or errors.

sdadams 4y ago
OpenBSD removed support for extended attributes.
- minusf 4y ago
  not a 1:1 equivalent of the "classic" extended attr but there are special flags on files. https://man.openbsd.org/chflags https://man.openbsd.org/chflags
- adrian_b 4y ago
  I suppose that this has been done to ensure less work for the OpenBSD maintainers, but I do not consider it as a wise decision. Extended file attributes can be used to implement a large number of useful things, many of them enhancing security, e.g. access-control lists. However, IIRC OpenBSD has chosen to also not implement ACLs. In any case regardless how useful or not useful extended file attributes are considered to be, deciding to not implement them in the main file system used by an operating system has the immediate consequence of disqualifying this operating system for the use on a file server (a.k.a. NAS), which is an application domain where the *BSD operating systems have been traditionally very good. The reason is that whenever such a NAS would have Windows, Linux or FreeBSD clients, transiting any file through that NAS would potentially lose data. In general, in the documentation of any file system, the lack of support for features like extended attributes or access-control lists must be displayed very prominently, to warn any potential users about the risk of data loss during copy operations (because the file copy commands are usually stupid enough to not inform the users whenever they are stripping file metadata, so such a loss may be discovered only when it is too late).
  - sdadams 4y ago
    In another project of mine I considered using extended attributes to tag files into categories. With extended attributes not being universally supported and easily overlooked the conclusion I came to was that I should store that information in a database file. I didn't want to one day lose my (manually assigned) tags to an erroneous move command. Anything non-native to OpenBSD can still be stored in muxfs as an archive file. You would need to ensure to pass the right arguments to the archiver to preserve the attributes in this case.

sdadams 4y ago

(1) I believe that stable inode numbers are possible with FUSE but they must be implemented by the FUSE driver. Mirroring over a network is not the intended use case. The mirrors exist to provide data redundancy for use by the muxfs driver. If you need to copy the data to a new location there is the muxfs sync command. (2) muxfs uses sequence numbers to count the write operations performed on each mirror. Upon failure to mount due to the mirrors being out of sync a report is printed comparing the first mirror with the first non-matching mirror, and this includes their sequence numbers.