4 ms·

> NFS is also more tolerant to breaking and resuming the TCP connection; this is fatal to sshfs. Except when... - The kernel driver crashes, resulting in unki

> NFS is also more tolerant to breaking and resuming the TCP connection; this is fatal to sshfs. Except when... - The kernel driver crashes, resulting in unkillable processes and orphaned, mounted files that, if accessed, result in those very processes also being unresponsive and unkillable. The solution? Reboot your machine. - There is packet loss on the network which can greatly compromise the stability of the NFS client. The solution? Don't have packet loss. - The RPC daemon somehow goes out of lockstep with the kernel, causing hanging and crashes. The solution? Don't let the RPC daemon get out of lockstep --- somehow. - You foolishly misconfigure the mount options, causing NFS to hang or time out if there's even a minor issue. Woe betide you if you muck up the timeout, retrans, etc. flags. The solution? Know the arcana and all your needs ahead of time, or be prepared for days of debugging. NFS sure is convenient and useful, but it's also terrible and antiquated.

__turbobrew__ 4y ago
Yea, NFS processes can get into an uninterruptible and unkillable sleep state where then only way to clean up is to reboot.
- bravetraveler 4y ago
  So glad/validated to see this thread. I always inevitably end up ranting about this when NFS comes up. The frustrating thing is, a shocking number of people either don't realize it or simply can't. I've lost weeks of my life to D-state. All TCP, all new protocol revisions. Datacenter wide deployment with redundant ports, paths, everything.
- chasil 4y ago
  Instead of rebooting, try a lazy unmount on Linux. umount -l /stuck/directory
  - vetrom 4y ago
    You still get stuck processes on D state which can be locking other resources. Maybe the kernel could use an alternate kill api for this, as being able to kill a D state process normally is problematic in many apps and should never be possible 'accidentally'.
- alexnewman 4y ago
  Soft mount no?
chasil 4y ago
You might be correct about NFSv2. However, there was significant exchange between NFSv4 and SMB versions 2 and 3. Neither is perfect, but they have practical uses. As far as packet loss is concerned, TCP is more robust, and is the default on Linux starting with NFSv3. When using NFSv4, rpc.portmap is not necessary, so it cannot "go out of lockstep with the kernel" because it is not running. As far as "foolish misconfiguration," solving a problem between the keyboard and the chair is not in the current features. Maybe the next release. p.s. NFS over UDP can cause silent corruption over gigabit ethernet - see "man 5 nfs" for details (search for "Using NFS over UDP on high-speed links"). This is one reason why TCP is the default.
- rollcat 4y ago
  > As far as packet loss is concerned, TCP is more robust Somehow I always felt using UDP as a basis for a file access protocol had more potential. You shouldn't care about packets arriving in the correct order, only that the (likely, concurrent) read/write operations are completed. If I request a 1MB read, I don't need it to come in 1500b sequential chunks, I can wait for the read call to return me a full 1MB buffer. It could also enable better throughput on high-latency (but otherwise speedy) links, like moving files between datacenters across the pond.
  - boulos 4y ago
    Sun RPC / XDR doesn't have consistency checking like that built in. So you definitely can't bolt this onto older NFS protocols, but they could have for NFS v4 (and didn't).
- vetrom 4y ago
  oof, the situation can be even rougher than that. At 9000 MTU on IPv4, you have a 'safe' NFS-over-UDP data rate of roughly 18 megabytes/second. Plenty of room at 1GbE, but not enough for 10GbE or greater. I suppose using krb5i/p mounts or some other integrity-providing GSSAPI/crypto implementation could help with this, but thats a different pile of worms. You would also get issues with not-strictly-necessary retransmits impacting your effective bandwidth. Thankfully, IPV6 can help mitigate this problem by extending the fragment id length from 16 to 32 bits. This gets you up to roughly 1.18 trillion bytes of 'safe' NFS traffic/second. There's plenty of reasons why IPv6 adoption is still fragmentary, but if you're running a high throughput network, this is certainly a motivator to move at least part of it to IPv6.

__turbobrew__ 4y ago

Yea, NFS processes can get into an uninterruptible and unkillable sleep state where then only way to clean up is to reboot.

bravetraveler 4y ago
So glad/validated to see this thread. I always inevitably end up ranting about this when NFS comes up. The frustrating thing is, a shocking number of people either don't realize it or simply can't. I've lost weeks of my life to D-state. All TCP, all new protocol revisions. Datacenter wide deployment with redundant ports, paths, everything.
chasil 4y ago
Instead of rebooting, try a lazy unmount on Linux. umount -l /stuck/directory
- vetrom 4y ago
  You still get stuck processes on D state which can be locking other resources. Maybe the kernel could use an alternate kill api for this, as being able to kill a D state process normally is problematic in many apps and should never be possible 'accidentally'.
alexnewman 4y ago
Soft mount no?

chasil 4y ago

You might be correct about NFSv2. However, there was significant exchange between NFSv4 and SMB versions 2 and 3. Neither is perfect, but they have practical uses. As far as packet loss is concerned, TCP is more robust, and is the default on Linux starting with NFSv3. When using NFSv4, rpc.portmap is not necessary, so it cannot "go out of lockstep with the kernel" because it is not running. As far as "foolish misconfiguration," solving a problem between the keyboard and the chair is not in the current features. Maybe the next release. p.s. NFS over UDP can cause silent corruption over gigabit ethernet - see "man 5 nfs" for details (search for "Using NFS over UDP on high-speed links"). This is one reason why TCP is the default.

rollcat 4y ago
> As far as packet loss is concerned, TCP is more robust Somehow I always felt using UDP as a basis for a file access protocol had more potential. You shouldn't care about packets arriving in the correct order, only that the (likely, concurrent) read/write operations are completed. If I request a 1MB read, I don't need it to come in 1500b sequential chunks, I can wait for the read call to return me a full 1MB buffer. It could also enable better throughput on high-latency (but otherwise speedy) links, like moving files between datacenters across the pond.
- boulos 4y ago
  Sun RPC / XDR doesn't have consistency checking like that built in. So you definitely can't bolt this onto older NFS protocols, but they could have for NFS v4 (and didn't).
vetrom 4y ago
oof, the situation can be even rougher than that. At 9000 MTU on IPv4, you have a 'safe' NFS-over-UDP data rate of roughly 18 megabytes/second. Plenty of room at 1GbE, but not enough for 10GbE or greater. I suppose using krb5i/p mounts or some other integrity-providing GSSAPI/crypto implementation could help with this, but thats a different pile of worms. You would also get issues with not-strictly-necessary retransmits impacting your effective bandwidth. Thankfully, IPV6 can help mitigate this problem by extending the fragment id length from 16 to 32 bits. This gets you up to roughly 1.18 trillion bytes of 'safe' NFS traffic/second. There's plenty of reasons why IPv6 adoption is still fragmentary, but if you're running a high throughput network, this is certainly a motivator to move at least part of it to IPv6.