4 ms·

What a coincidence! For a project that I maintain (Buildbarn, a distributed build cluster for Bazel) I recently generalized all of the FUSE code I had into a ge

What a coincidence! For a project that I maintain (Buildbarn, a distributed build cluster for Bazel) I recently generalized all of the FUSE code I had into a generic VFS that can both be exposed over FUSE and NFSv4. The intent was the same: to provide a better out of the box experience on macOS. Here's a design doc I wrote on that change. Slight warning that it's written with some Buildbarn knowledge in mind. https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.md https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.... Fortunately, fuse-t doesn't make any of my work unnecessary. Buildbarn uses go-fuse, which talks to the FUSE character directly instead of using libfuse. fuse-t would thus not be a drop-in replacement. Phew! PS: A bit unfortunate that fuse-t isn't Open Source. :-(

EdSchouten 4y ago
Replying to my own comment: I do wonder how this library deals with some of the fundamental differences beween FUSE and NFSv4. For example, with FUSE the kernel and server share intimate knowledge on which part of the file system lives in the kernel's inode cache. Only when the kernel issues a FORGET call, may the server drop information corresponding to a given nodeid. As NFSv4 is designed to be stateless, servers may need to be able to process requests containing arbitrary file handles, regardless of how long ago they were returned as part of some prior request. You therefore see that in-kernel implementations of NFS servers rely on special file system methods to resolve objects by file handle, in addition to being able to resolve by path. This means that if you implement FUSE on top of NFSv4, you will most likely not be able to purge any state. Your FUSE file system's FORGET method will probably never be called. This means that memory usage of fuse-t will most likely just keep on growing as time progresses? Or it announces itself as using file handle mode FH4_VOLATILE_*, but UNIX-like NFSv4 clients hardly ever know how to deal with that.
- boulos 4y ago
  Nit: NFSv3 is stateless, V4 isn't (sessionid and clientid permit stateful handling of locks and shares). I would assume this person is just doing an in-memory "NFS server" that would keep all of that state around. So it's more like a FUSE-compatible layer that speaks NFSv4 as a "front end" (since NFS clients are better than "ha ha, surprise! This FUSE backend is networked and can fail on weird ways"). I'm not sure how they chose to translate things like DELEGATE, SEQUENCE, and so on (probably just reply with an error). But for basic OPEN, READ, WRITE, etc. it's all fairly straightforward. I had hoped they used nfs-ganesha for the NFS server / frontend, but the attributions file suggests they probably rolled their own (and thus is going to be full of bugs and immature for quite some time).
  - EdSchouten 4y ago
    With it being stateless, I was referring to just the part where I wanted to point out the difference: file handles are stateless, while FUSE's equivalent (nodeids) are stateful. Regardless of whether it's an in-memory or persistent solution, the problem remains: fuse-t has little choice but to leak resources of the underlying FUSE file system, as there is no valid point in time in which you can issue FORGET operations. This means that any file system with some form of churn rate will leak memory. Also note that delegation is effectively optional. If a server simply always replies with OPEN_DELEGATE_NONE, the client has to interact with the file as if it's stored remotely. There is no need to implement SEQUENCE, by the way. macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.
    - concerned_ctzn 4y ago
      I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. Besides, how many FUSE filesystems implement FORGET? Anyway this is the reason I went with NFSv4. I started the project with NFS v3 but then discovered many limitations: being stateless, no named attributes, questionable locking support, etc. So Eventually I dropped it and re-implemented everything on NVSv4.
      - EdSchouten 4y ago
        > I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. As long as the file the NFSv4 file handle refers to is still usable (i.e., linked into the file system), the NFSv4 file handle must remain usable. Note that this is not a universal requirement, but at least one that the macOS NFSv4 client enforces. It only implements FH4_PERSISTENT. This doesn't seem to be documented explicitly, but is somewhat revealed by this printf(): https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vfsops.c#L2734-L2736 https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vf... > Besides, how many FUSE filesystems implement FORGET? Any file system that wants to remove files in the background (meaning: not by calling unlink() through the FUSE mount) must likely do proper refcounting on such files, and provide an implementation of FORGET. For example, a file system that can give information on live football matches may want to remove files/directories belonging to matches that have already ended. In that case you want the node IDs to remain valid, but refer to files that have already been unlinked from the file system hierarchy. The FORGET operation allows you to determine when those files can be removed from the FUSE server's bookkeeping entirely. Here is an implementation of FORGET that I wrote for Buildbarn: https://github.com/buildbarn/bb-remote-execution/blob/master/pkg/filesystem/virtual/fuse/simple_raw_file_system.go#L267-L297 https://github.com/buildbarn/bb-remote-execution/blob/master...
- fulafel 4y ago
  The DB for the necessary state could also be on-disk. Maybe in practice you can expire things after a reasonably long time, like a week, even though that's not strictly spec conforming.
justsomeadvice0 4y ago
You might look at webdav instead, which has actual existing library implementations (unlike NFSv4), supports user/pass auth (in some cases you don't want wide-open services on loopback) and is based on HTTP; that would make the whole thing easier. Various projects offer a webdav combat layer, see cryptomator for one example: https://docs.cryptomator.org/en/latest/desktop/vault-mounting/ https://docs.cryptomator.org/en/latest/desktop/vault-mountin... I went down the NFS road long ago - I somewhat (??) remember that coaxing macOS to mount_nfs on loopback is actually a PITA (I remember having to activate 127.0.0.* and mount 127.0.0.2).
- skissane 4y ago
  I remember, around 10 years ago, mucking around with WebDAV, and discovering many WebDAV clients were extremely crapulent. Especially on Windows. Maybe, some years later, Windows' WebDAV client has improved. And I can't remember whether macOS WebDAV was better or not.
  - deleted 4y ago
    [deleted]
- EdSchouten 4y ago
  For Buildbarn I need to provide a writable file system that acts in a POSIX compliant way. I'm not convinced WebDAV could provide that.
  - justsomeadvice0 4y ago
    Fair point, agreed I doubt mount_webdav ends up posix compliant.

EdSchouten 4y ago

Replying to my own comment: I do wonder how this library deals with some of the fundamental differences beween FUSE and NFSv4. For example, with FUSE the kernel and server share intimate knowledge on which part of the file system lives in the kernel's inode cache. Only when the kernel issues a FORGET call, may the server drop information corresponding to a given nodeid. As NFSv4 is designed to be stateless, servers may need to be able to process requests containing arbitrary file handles, regardless of how long ago they were returned as part of some prior request. You therefore see that in-kernel implementations of NFS servers rely on special file system methods to resolve objects by file handle, in addition to being able to resolve by path. This means that if you implement FUSE on top of NFSv4, you will most likely not be able to purge any state. Your FUSE file system's FORGET method will probably never be called. This means that memory usage of fuse-t will most likely just keep on growing as time progresses? Or it announces itself as using file handle mode FH4_VOLATILE_*, but UNIX-like NFSv4 clients hardly ever know how to deal with that.

boulos 4y ago
Nit: NFSv3 is stateless, V4 isn't (sessionid and clientid permit stateful handling of locks and shares). I would assume this person is just doing an in-memory "NFS server" that would keep all of that state around. So it's more like a FUSE-compatible layer that speaks NFSv4 as a "front end" (since NFS clients are better than "ha ha, surprise! This FUSE backend is networked and can fail on weird ways"). I'm not sure how they chose to translate things like DELEGATE, SEQUENCE, and so on (probably just reply with an error). But for basic OPEN, READ, WRITE, etc. it's all fairly straightforward. I had hoped they used nfs-ganesha for the NFS server / frontend, but the attributions file suggests they probably rolled their own (and thus is going to be full of bugs and immature for quite some time).
- EdSchouten 4y ago
  With it being stateless, I was referring to just the part where I wanted to point out the difference: file handles are stateless, while FUSE's equivalent (nodeids) are stateful. Regardless of whether it's an in-memory or persistent solution, the problem remains: fuse-t has little choice but to leak resources of the underlying FUSE file system, as there is no valid point in time in which you can issue FORGET operations. This means that any file system with some form of churn rate will leak memory. Also note that delegation is effectively optional. If a server simply always replies with OPEN_DELEGATE_NONE, the client has to interact with the file as if it's stored remotely. There is no need to implement SEQUENCE, by the way. macOS implements NFSv4.0, while SEQUENCE is part of NFSv4.1 and later.
  - concerned_ctzn 4y ago
    I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. Besides, how many FUSE filesystems implement FORGET? Anyway this is the reason I went with NFSv4. I started the project with NFS v3 but then discovered many limitations: being stateless, no named attributes, questionable locking support, etc. So Eventually I dropped it and re-implemented everything on NVSv4.
    - EdSchouten 4y ago
      > I don't understand the comment about leaking memory, nfs file handles don't have to be persistent. As long as the file the NFSv4 file handle refers to is still usable (i.e., linked into the file system), the NFSv4 file handle must remain usable. Note that this is not a universal requirement, but at least one that the macOS NFSv4 client enforces. It only implements FH4_PERSISTENT. This doesn't seem to be documented explicitly, but is somewhat revealed by this printf(): https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vfsops.c#L2734-L2736 https://github.com/apple/darwin-xnu/blob/main/bsd/nfs/nfs_vf... > Besides, how many FUSE filesystems implement FORGET? Any file system that wants to remove files in the background (meaning: not by calling unlink() through the FUSE mount) must likely do proper refcounting on such files, and provide an implementation of FORGET. For example, a file system that can give information on live football matches may want to remove files/directories belonging to matches that have already ended. In that case you want the node IDs to remain valid, but refer to files that have already been unlinked from the file system hierarchy. The FORGET operation allows you to determine when those files can be removed from the FUSE server's bookkeeping entirely. Here is an implementation of FORGET that I wrote for Buildbarn: https://github.com/buildbarn/bb-remote-execution/blob/master/pkg/filesystem/virtual/fuse/simple_raw_file_system.go#L267-L297 https://github.com/buildbarn/bb-remote-execution/blob/master...
fulafel 4y ago
The DB for the necessary state could also be on-disk. Maybe in practice you can expire things after a reasonably long time, like a week, even though that's not strictly spec conforming.

justsomeadvice0 4y ago

You might look at webdav instead, which has actual existing library implementations (unlike NFSv4), supports user/pass auth (in some cases you don't want wide-open services on loopback) and is based on HTTP; that would make the whole thing easier. Various projects offer a webdav combat layer, see cryptomator for one example: https://docs.cryptomator.org/en/latest/desktop/vault-mounting/ https://docs.cryptomator.org/en/latest/desktop/vault-mountin... I went down the NFS road long ago - I somewhat (??) remember that coaxing macOS to mount_nfs on loopback is actually a PITA (I remember having to activate 127.0.0.* and mount 127.0.0.2).

skissane 4y ago
I remember, around 10 years ago, mucking around with WebDAV, and discovering many WebDAV clients were extremely crapulent. Especially on Windows. Maybe, some years later, Windows' WebDAV client has improved. And I can't remember whether macOS WebDAV was better or not.
- deleted 4y ago
  [deleted]
EdSchouten 4y ago
For Buildbarn I need to provide a writable file system that acts in a POSIX compliant way. I'm not convinced WebDAV could provide that.
- justsomeadvice0 4y ago
  Fair point, agreed I doubt mount_webdav ends up posix compliant.