6 ms·

Willscott/go-NFS: Golang NFSv3 server

137 points by pjf 6y ago

rawoke083600 6y ago
Good work ! PS is NFS still a thing ? I thought Gluster(FS) or S3 sorta replaced it in most shops ? I could be mistaken of course.
- zuppy 6y ago
  I’m using nfs for Docker on Mac, as its native share system (osxfs) has awful performance. Not the best setup, but didn’t find a better one yet.
  - rawoke083600 6y ago
    Interesting... maybe give Gluster a spin when you bored of have time.
cyphar 6y ago
When looking for something like this a year or so ago, I found [1] which supports both NFSv3 and v4 as well as p9. It worked alright in my experience though I eventually switched to ZFS which has built in support to auto-configure NFS shares. [1]: https://github.com/nfs-ganesha/nfs-ganesha https://github.com/nfs-ganesha/nfs-ganesha
Zach_the_Lizard 6y ago
I wonder if this is a good way to support projects like Microsoft's Git VFS without requiring specific kernel drivers. Mounting a special Git NFS drive + having all the magic hidden behind NFS seems like it could be interesting.
willscott 6y ago
Wasn't expecting to see this here! This turned out easier than i was expecting it to be. It's nice to be able to mount a VFS without needing privileges on the server side. The main intention for this code is to eventually use it to replace fuse on mac, since nfs is a valid mount time for mac clients to consume.
- _prometheus 6y ago
  This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs. Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.
  - jchw 6y ago
    Just to lay them out, some notable other options: - Under Linux and FreeBSD, bazil fuse offers a nice low dependency path to supporting FUSE. It is all pure Go code. - Under Windows, you can use WinFSP, which is a rock solid usermode filesystem driver with FUSE support. Despite its name, you can use this with Cgofuse to get FUSE on Windows without CGo. Combining the two above gets you pretty far already (though if you want to actually avoid a CGo dependency you do need to explicitly disable it.) I think the next level would be to have a general filesystem interface (there are several options in the ecosystem already) that can be used as a common ground between different usermode filesystem and network filesystem server implementations. Then ideally, a magic Go library can exist that gives you the best possible setup with switchable backends depending on the platform and build configuration. Right now Go itself is possibly standardizing a read-only filesystem interface, an extension of this with write support might serve as a decent jumping point for getting toward an idealized pluggable filesystem library.
    - yencabulator 6y ago
      > I think the next level would be to have a general filesystem interface That's starting to take shape: https://old.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/ https://old.reddit.com/r/golang/comments/hv976o/qa_iofs_draf... It won't be enough for the likes of FUSE, though. Some extensions might be enough to bridge that gap, it's too early to tell. FUSE is pretty particular about things like rename(2) preserving node identity.
  - adinisom 6y ago
    Another option: Fairmount, an app DVD ripping on MacOS, uses WebDAV: https://github.com/BoxOfSnoo/Fairmount https://github.com/BoxOfSnoo/Fairmount
pbnjay 6y ago
I've always wondered if I could take the Unix "everything is a file" approach way too far and hook it up to a web service. This looks like exactly the kind of glue to make that easy...
- maxmcd 6y ago
  Here's a thing like that for youtube: https://github.com/rasguanabana/ytfs https://github.com/rasguanabana/ytfs
- nitrogen 6y ago
  My favorite "everything is a filesystem" hack from around 2001 was cdfs for Linux, which would expose each audio track as a .wav file you could cp, and each separate data session as a subdirectory, so you could access prior states of the CD filesystem.
brianm 6y ago
This is cool! At a previous company we got a lot of mileage in debugging and ad-hoc tasks by exposing various things as filesystems. We mostly did webdav, but NFS is way better from a client transparency perspective. For many folks find, ls, and grep very much beat curl and jq.
- tomcam 6y ago
  I would love to understand more about what you exposed as filesystems and why. Is that on https://skife.org/ https://skife.org/ somewhere?
- q3k 6y ago
  A nice protocol to do this with is 9p [1]. It's much easier to implement a server for it than for NFS, either from scratch or even with a good library, and it's about just as usable on major operating systems. It has a tiny API and therefore surface area, also making it great for high security applications. It’s probably the closest thing we have to a cross-platform, network transportable FUSE. Even though it started its life as a component of Plan9, 9p is getting through a renaissance period right now: qemu, WSL, gVisor, ChromeOS VMs (crostini)... all use 9p! [1] - https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol)
  - trasz 6y ago
    9P is fine as kind of lowest common denominator network filesystem, as long as you don’t hit its fundamental limitations, such as not being particularly Unix-compatible, semantics wise.
  - ComputerGuru 6y ago
    Unfortunately all the default implementations really suck at performance (for Linux and Windows, at any rate), and it’s very noticeable under IO pressure, lots of small files, etc. They're stable and seem to be correct though, which is something.
toomuchtodo 6y ago
@willscott: Do you have any experience to share running this over Wireguard or similar? Edit: Thank you!
- willscott 6y ago
  Most testing has been on a Lan or on the local machine for local VFS use cases. This layer eschews responsibility for multiple concurrent clients for simplicity. No promises you'll get either decent performance or proper cache invalidation when using it that way. It tends to be conservative in not filling in all the opportunistic caches. There's a hook the backing application can use to tune reads and write sizes that becomes much more relevant in a network case. How those should be set in practice isn't something I've spent time on.
anderspitman 6y ago
Considering how many things have been implemented on top of HTTP, I find it interesting that something like WebDAV (or Solid[0], or remoteStorage[1]) hasn't nearly completely supplanted NFS/SFTP/etc. Not saying that would be ideal from a technical perspective (WebDAV at least has some issues), just surprised the ability to access remote filesystems from the browser hasn't been a bigger driver. For example, there are a lot of interesting apps built on top of Google Drive as the storage backend, but overall the concept doesn't seem to have gained much traction. [0]: https://inrupt.com/solid https://inrupt.com/solid [1]: https://remotestorage.io/ https://remotestorage.io/
- mprovost 6y ago
  Sun developed WebNFS [0] specifically so Java applets (remember them?) could access remote filesystems through firewalls from the browser. It's basically the same protocol as NFSv3 with some clever workarounds so that it doesn't require the separate MOUNT protocol to set up a connection (or the portmapper). That way everything runs on a single port. [0] https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS
- protomyth 6y ago
  Because WebDAV implementations aren't that good and a true pain in the butt for system admins to setup. If someone made one that was half as easy to setup as say PHP, it would be doing really well right now.
- api 6y ago
  NFS is a lot faster. A custom protocol will almost always beat a kludge on top of HTTP
  - fulafel 6y ago
    But NFS isn't a custom protocol for any specific app either, and apps that need good fs performance (eg databases, video editing platforms, "big data" compute platforms etc) avoid it. Implementation-wise HTTP has the advantage that the apps can tune the client code because the protocol client implementation isn't in the kernel. In addition HTTP security (transport & user authentication) is works and everyone understands it, whereas on NFS it's a huge mess and enemy of interoperability. In practical popularity NFS fares extremely poorly eg on AWS (how many apps use EFS vs S3).
anderspitman 6y ago
Didn't see it mentioned in the README, so I'll ask here: any particular reason to go NFSv3 vs NFSv4? I'm not familiar enough with the protocol to venture an educated guess.
- willscott 6y ago
  V4 has many more things going on (it's stateful, has a much more complicated locking system integrated, and adds some other points of complexity that get better performance). More than I was ready to take on for a first pass :)
  - anderspitman 6y ago
    You had me at stateful
    - yencabulator 6y ago
      For what it's worth, stateful is a good thing; NFSv4 leases are what allows it to safely do more aggressive caching than NFSv3. Back when it was new, benchmarks showed a good speed increase all around...
      - anderspitman 6y ago
        Stateful isn't inherently good or bad. It's a tradeoff. But the more programming I do the less I think it's a tradeoff worth making.
        yencabulator 6y ago
        In the context of NFS, it's a trade-off that enabled higher performance (and some new features too).
jonathonf 6y ago
Possibly useful existing implementation: https://github.com/unfs3/unfs3 https://github.com/unfs3/unfs3 This has the benefits of being tested and having a working read-write implementation (along with still being user-space).

137 points by pjf 6y ago

Good work ! PS is NFS still a thing ? I thought Gluster(FS) or S3 sorta replaced it in most shops ? I could be mistaken of course.

zuppy 6y ago
I’m using nfs for Docker on Mac, as its native share system (osxfs) has awful performance. Not the best setup, but didn’t find a better one yet.
- rawoke083600 6y ago
  Interesting... maybe give Gluster a spin when you bored of have time.

cyphar 6y ago

When looking for something like this a year or so ago, I found [1] which supports both NFSv3 and v4 as well as p9. It worked alright in my experience though I eventually switched to ZFS which has built in support to auto-configure NFS shares. [1]: https://github.com/nfs-ganesha/nfs-ganesha https://github.com/nfs-ganesha/nfs-ganesha

Zach_the_Lizard 6y ago

I wonder if this is a good way to support projects like Microsoft's Git VFS without requiring specific kernel drivers. Mounting a special Git NFS drive + having all the magic hidden behind NFS seems like it could be interesting.

willscott 6y ago

Wasn't expecting to see this here! This turned out easier than i was expecting it to be. It's nice to be able to mount a VFS without needing privileges on the server side. The main intention for this code is to eventually use it to replace fuse on mac, since nfs is a valid mount time for mac clients to consume.

_prometheus 6y ago
This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs. Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.
- jchw 6y ago
  Just to lay them out, some notable other options: - Under Linux and FreeBSD, bazil fuse offers a nice low dependency path to supporting FUSE. It is all pure Go code. - Under Windows, you can use WinFSP, which is a rock solid usermode filesystem driver with FUSE support. Despite its name, you can use this with Cgofuse to get FUSE on Windows without CGo. Combining the two above gets you pretty far already (though if you want to actually avoid a CGo dependency you do need to explicitly disable it.) I think the next level would be to have a general filesystem interface (there are several options in the ecosystem already) that can be used as a common ground between different usermode filesystem and network filesystem server implementations. Then ideally, a magic Go library can exist that gives you the best possible setup with switchable backends depending on the platform and build configuration. Right now Go itself is possibly standardizing a read-only filesystem interface, an extension of this with write support might serve as a decent jumping point for getting toward an idealized pluggable filesystem library.
  - yencabulator 6y ago
    > I think the next level would be to have a general filesystem interface That's starting to take shape: https://old.reddit.com/r/golang/comments/hv976o/qa_iofs_draft_design/ https://old.reddit.com/r/golang/comments/hv976o/qa_iofs_draf... It won't be enough for the likes of FUSE, though. Some extensions might be enough to bridge that gap, it's too early to tell. FUSE is pretty particular about things like rename(2) preserving node identity.
- adinisom 6y ago
  Another option: Fairmount, an app DVD ripping on MacOS, uses WebDAV: https://github.com/BoxOfSnoo/Fairmount https://github.com/BoxOfSnoo/Fairmount

pbnjay 6y ago

I've always wondered if I could take the Unix "everything is a file" approach way too far and hook it up to a web service. This looks like exactly the kind of glue to make that easy...

maxmcd 6y ago
Here's a thing like that for youtube: https://github.com/rasguanabana/ytfs https://github.com/rasguanabana/ytfs
nitrogen 6y ago
My favorite "everything is a filesystem" hack from around 2001 was cdfs for Linux, which would expose each audio track as a .wav file you could cp, and each separate data session as a subdirectory, so you could access prior states of the CD filesystem.

brianm 6y ago

This is cool! At a previous company we got a lot of mileage in debugging and ad-hoc tasks by exposing various things as filesystems. We mostly did webdav, but NFS is way better from a client transparency perspective. For many folks find, ls, and grep very much beat curl and jq.

tomcam 6y ago
I would love to understand more about what you exposed as filesystems and why. Is that on https://skife.org/ https://skife.org/ somewhere?
q3k 6y ago
A nice protocol to do this with is 9p [1]. It's much easier to implement a server for it than for NFS, either from scratch or even with a good library, and it's about just as usable on major operating systems. It has a tiny API and therefore surface area, also making it great for high security applications. It’s probably the closest thing we have to a cross-platform, network transportable FUSE. Even though it started its life as a component of Plan9, 9p is getting through a renaissance period right now: qemu, WSL, gVisor, ChromeOS VMs (crostini)... all use 9p! [1] - https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol)
- trasz 6y ago
  9P is fine as kind of lowest common denominator network filesystem, as long as you don’t hit its fundamental limitations, such as not being particularly Unix-compatible, semantics wise.
- ComputerGuru 6y ago
  Unfortunately all the default implementations really suck at performance (for Linux and Windows, at any rate), and it’s very noticeable under IO pressure, lots of small files, etc. They're stable and seem to be correct though, which is something.

toomuchtodo 6y ago

@willscott: Do you have any experience to share running this over Wireguard or similar? Edit: Thank you!

willscott 6y ago
Most testing has been on a Lan or on the local machine for local VFS use cases. This layer eschews responsibility for multiple concurrent clients for simplicity. No promises you'll get either decent performance or proper cache invalidation when using it that way. It tends to be conservative in not filling in all the opportunistic caches. There's a hook the backing application can use to tune reads and write sizes that becomes much more relevant in a network case. How those should be set in practice isn't something I've spent time on.

anderspitman 6y ago

Considering how many things have been implemented on top of HTTP, I find it interesting that something like WebDAV (or Solid[0], or remoteStorage[1]) hasn't nearly completely supplanted NFS/SFTP/etc. Not saying that would be ideal from a technical perspective (WebDAV at least has some issues), just surprised the ability to access remote filesystems from the browser hasn't been a bigger driver. For example, there are a lot of interesting apps built on top of Google Drive as the storage backend, but overall the concept doesn't seem to have gained much traction. [0]: https://inrupt.com/solid https://inrupt.com/solid [1]: https://remotestorage.io/ https://remotestorage.io/

mprovost 6y ago
Sun developed WebNFS [0] specifically so Java applets (remember them?) could access remote filesystems through firewalls from the browser. It's basically the same protocol as NFSv3 with some clever workarounds so that it doesn't require the separate MOUNT protocol to set up a connection (or the portmapper). That way everything runs on a single port. [0] https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS
protomyth 6y ago
Because WebDAV implementations aren't that good and a true pain in the butt for system admins to setup. If someone made one that was half as easy to setup as say PHP, it would be doing really well right now.
api 6y ago
NFS is a lot faster. A custom protocol will almost always beat a kludge on top of HTTP
- fulafel 6y ago
  But NFS isn't a custom protocol for any specific app either, and apps that need good fs performance (eg databases, video editing platforms, "big data" compute platforms etc) avoid it. Implementation-wise HTTP has the advantage that the apps can tune the client code because the protocol client implementation isn't in the kernel. In addition HTTP security (transport & user authentication) is works and everyone understands it, whereas on NFS it's a huge mess and enemy of interoperability. In practical popularity NFS fares extremely poorly eg on AWS (how many apps use EFS vs S3).

anderspitman 6y ago

Didn't see it mentioned in the README, so I'll ask here: any particular reason to go NFSv3 vs NFSv4? I'm not familiar enough with the protocol to venture an educated guess.

willscott 6y ago
V4 has many more things going on (it's stateful, has a much more complicated locking system integrated, and adds some other points of complexity that get better performance). More than I was ready to take on for a first pass :)
- anderspitman 6y ago
  You had me at stateful
  - yencabulator 6y ago
    For what it's worth, stateful is a good thing; NFSv4 leases are what allows it to safely do more aggressive caching than NFSv3. Back when it was new, benchmarks showed a good speed increase all around...
    - anderspitman 6y ago
      Stateful isn't inherently good or bad. It's a tradeoff. But the more programming I do the less I think it's a tradeoff worth making.
      - yencabulator 6y ago
        In the context of NFS, it's a trade-off that enabled higher performance (and some new features too).

jonathonf 6y ago

Possibly useful existing implementation: https://github.com/unfs3/unfs3 https://github.com/unfs3/unfs3 This has the benefits of being tested and having a working read-write implementation (along with still being user-space).