4 ms·

When the choice is between "slow" file transfers and being unable to do file transfers because of NFS timeouts, the choice should be obvious. I've lost data as

by Rygian 3y ago

When the choice is between "slow" file transfers and being unable to do file transfers because of NFS timeouts, the choice should be obvious. I've lost data as a side effect of a simple file transfer timing out.

suprjami 3y ago
Just the small amount of people using NFS suggests this tunable should remain the default. Nothing is stopping sysadmins tuning for their environment. There's no one-size-fits-all answer, which is why it's a tunable.
- tremon 3y ago
  This isn't just about NFS timeouts. Try playing a movie from a rotational disk while simultaneously doing high-volume writes. You will get frequent pauses in your video because the write buffer size is so large that a single writeback will cause the video buffer to drain empty. On my desktop with 32GB ram, I can even get audio to skip when ripping DVD's to disk. That's because practically the entire movie fits into ram before Linux decides to start the writeback process, and that writeback process will hog the disk for almost a minute. Or it used to, until I reduced the buffer size by a full order of magnitude. This is just another sad example of buffer bloat: the inability to tune data buffers to the capacity of the underlying stream.
  - koverstreet 3y ago
    It's also because doing real IO scheduling has gotten a lot harder with fast NVME devices and blk-mq. It's a real problem.
  - bcook 3y ago
    Have you played with "ionice"? Did it help?
  - M95D 3y ago
    That's another thing I can't understand: Why does NFS timeout when the data transfer is still on? Shouldn't it timeout only when the server is no longer ACK-ing packets?
    - Tyrannosaur 3y ago
      > Shouldn't it timeout only when the server is no longer ACK-ing packets? That's exactly what happens. The server ACKs data until it fills its write buffer, and then stalls unresponsive until the entire buffer is flushed to disk. If it takes longer to flush the buffer to disk than the client's timeout, it gives up. I have personally watched this happen via wireshark where the server doesn't ACK for more than 10 minutes.
      - CableNinja 3y ago
        Holy shit i think you and the above comment, along with this thread, may have finally given me the answer to one of the few problems i was never able to solve. About 4-5 years ago, i was working on a project, and part of that was copying big amounts of data to a system via nfs. At 30 minutes exactly, nfs would croak, transfer fails. I think this buffer fill and empty flow was fucking killing it. Its a shame i dont work there anymore, id definitely wanna try tweaking these settings and see if i could solve it
        Tyrannosaur 3y ago
        Yeah that does sound like the symptoms of the problem I discovered. If you ever witness it again, the trick is watching /proc/meminfo for the Dirty and Writeback numbers. And it's the vm.dirty* settings to change to fix it as described here: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ https://lonesysadmin.net/2013/12/22/better-linux-disk-cachin...
      - M95D 3y ago
        That's not it. I only had this problem on a fast-ethernet connection (because I had to share the cable for two connections). The server could write ~ 50 MB/s, but it still timed out on the 10MB/s upload.

by Rygian 3y ago

suprjami 3y ago

Just the small amount of people using NFS suggests this tunable should remain the default. Nothing is stopping sysadmins tuning for their environment. There's no one-size-fits-all answer, which is why it's a tunable.

tremon 3y ago
This isn't just about NFS timeouts. Try playing a movie from a rotational disk while simultaneously doing high-volume writes. You will get frequent pauses in your video because the write buffer size is so large that a single writeback will cause the video buffer to drain empty. On my desktop with 32GB ram, I can even get audio to skip when ripping DVD's to disk. That's because practically the entire movie fits into ram before Linux decides to start the writeback process, and that writeback process will hog the disk for almost a minute. Or it used to, until I reduced the buffer size by a full order of magnitude. This is just another sad example of buffer bloat: the inability to tune data buffers to the capacity of the underlying stream.
- koverstreet 3y ago
  It's also because doing real IO scheduling has gotten a lot harder with fast NVME devices and blk-mq. It's a real problem.
- bcook 3y ago
  Have you played with "ionice"? Did it help?
- M95D 3y ago
  That's another thing I can't understand: Why does NFS timeout when the data transfer is still on? Shouldn't it timeout only when the server is no longer ACK-ing packets?
  - Tyrannosaur 3y ago
    > Shouldn't it timeout only when the server is no longer ACK-ing packets? That's exactly what happens. The server ACKs data until it fills its write buffer, and then stalls unresponsive until the entire buffer is flushed to disk. If it takes longer to flush the buffer to disk than the client's timeout, it gives up. I have personally watched this happen via wireshark where the server doesn't ACK for more than 10 minutes.
    - CableNinja 3y ago
      Holy shit i think you and the above comment, along with this thread, may have finally given me the answer to one of the few problems i was never able to solve. About 4-5 years ago, i was working on a project, and part of that was copying big amounts of data to a system via nfs. At 30 minutes exactly, nfs would croak, transfer fails. I think this buffer fill and empty flow was fucking killing it. Its a shame i dont work there anymore, id definitely wanna try tweaking these settings and see if i could solve it
      - Tyrannosaur 3y ago
        Yeah that does sound like the symptoms of the problem I discovered. If you ever witness it again, the trick is watching /proc/meminfo for the Dirty and Writeback numbers. And it's the vm.dirty* settings to change to fix it as described here: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ https://lonesysadmin.net/2013/12/22/better-linux-disk-cachin...
    - M95D 3y ago
      That's not it. I only had this problem on a fast-ethernet connection (because I had to share the cable for two connections). The server could write ~ 50 MB/s, but it still timed out on the 10MB/s upload.