4 ms·

Any word on the iops numbers vs the in-kernel NFS client? The throughput is impressive but IME it ends up being the stat/fd activity of NFS clients that's the t

Any word on the iops numbers vs the in-kernel NFS client? The throughput is impressive but IME it ends up being the stat/fd activity of NFS clients that's the the limiting factor (try running `ls -l` in an NFS directory with lots of files in it, even worse if there are lots of symlinks involved.)

diamondlovesyou 5y ago
Presumably, with async you could queue all the `stat`-ops after the dir walk at once, leaving the total latency in the ball park of "dir walk"+"ping for file stat"+"ping remaining symlink stats if present". But I don't think `ls` does this. Otherwise, yeah, you incur network latency on every file, plus, as you say, symlink "pings" if those are present. So its "dir walk"+"number of files"*"ping"+"symlinks"*"ping", which adds up. Batching high-latency ops is one of the only cases where I like async.
- mprovost 5y ago
  NFS version 3 already has the READDIRPLUS operation which returns the directory contents and all of the stats together in one call to improve performance for this case. Sometimes the kernel client doesn't use READDIRPLUS and falls back to issuing a bunch of requests - usually because it's being super conservative about security (as if there is such a thing in NFS). It's also not that straightforward for a utility like ls to tell the kernel that it's reading both the directory and the stats for each file - they're separate system calls and the kernel has to figure out what the program is trying to do and optimise. I wrote a tool that issues raw READDIRPLUS requests to list a directory: https://rawgit.com/mprovost/NFStash/master/man/nfsls.8.html https://rawgit.com/mprovost/NFStash/master/man/nfsls.8.html
  - ninkendo 5y ago
    Wow, thanks! I’m definitely going to try this tool. I have use cases at work where I need to look at files in an NFS server on the other side of the country from my home desktop (don’t ask… it’s much bigger PITA to get SSH access to a VM near the NFS server than it should be) and it takes ages to run ls in a directory.
tarasglek 5y ago
Usually going via kernel nfs client will use up more memory bandwidth. I would expect lower per-client numbers. From what I've read you go from 3 memcopies on userspace to 4 with kernel nfs. I haven't yet instrumented memory bandwidth on my amd machines, but it feels like I'm at the limit.
tarasglek 5y ago
In this case my job was to generate a synthetic workload to drive max bandwidth. So I didn't have to worry about metadata. Metadata is more complicated to model, need realistic directory structures etc. https://www.spec.org/sfs2014/ https://www.spec.org/sfs2014/ (2020 version) is a good test of a metadata-heavy workloads, but isn't open source like fio :(.
- mprovost 5y ago
  The nice thing about NFS (v3) is that it's stateless, so you can just keep doing the same thing over and over. Messing around with metadata sucks but once you get the filehandle of the file that you want to read or write then you don't need it anymore. I wrote a suite of tools that does all this and dumps the NFS transactions as JSON, maybe it can be useful: https://github.com/mprovost/NFStash https://github.com/mprovost/NFStash

diamondlovesyou 5y ago

Presumably, with async you could queue all the `stat`-ops after the dir walk at once, leaving the total latency in the ball park of "dir walk"+"ping for file stat"+"ping remaining symlink stats if present". But I don't think `ls` does this. Otherwise, yeah, you incur network latency on every file, plus, as you say, symlink "pings" if those are present. So its "dir walk"+"number of files"*"ping"+"symlinks"*"ping", which adds up. Batching high-latency ops is one of the only cases where I like async.

mprovost 5y ago
NFS version 3 already has the READDIRPLUS operation which returns the directory contents and all of the stats together in one call to improve performance for this case. Sometimes the kernel client doesn't use READDIRPLUS and falls back to issuing a bunch of requests - usually because it's being super conservative about security (as if there is such a thing in NFS). It's also not that straightforward for a utility like ls to tell the kernel that it's reading both the directory and the stats for each file - they're separate system calls and the kernel has to figure out what the program is trying to do and optimise. I wrote a tool that issues raw READDIRPLUS requests to list a directory: https://rawgit.com/mprovost/NFStash/master/man/nfsls.8.html https://rawgit.com/mprovost/NFStash/master/man/nfsls.8.html
- ninkendo 5y ago
  Wow, thanks! I’m definitely going to try this tool. I have use cases at work where I need to look at files in an NFS server on the other side of the country from my home desktop (don’t ask… it’s much bigger PITA to get SSH access to a VM near the NFS server than it should be) and it takes ages to run ls in a directory.

tarasglek 5y ago

Usually going via kernel nfs client will use up more memory bandwidth. I would expect lower per-client numbers. From what I've read you go from 3 memcopies on userspace to 4 with kernel nfs. I haven't yet instrumented memory bandwidth on my amd machines, but it feels like I'm at the limit.

tarasglek 5y ago

In this case my job was to generate a synthetic workload to drive max bandwidth. So I didn't have to worry about metadata. Metadata is more complicated to model, need realistic directory structures etc. https://www.spec.org/sfs2014/ https://www.spec.org/sfs2014/ (2020 version) is a good test of a metadata-heavy workloads, but isn't open source like fio :(.

mprovost 5y ago
The nice thing about NFS (v3) is that it's stateless, so you can just keep doing the same thing over and over. Messing around with metadata sucks but once you get the filehandle of the file that you want to read or write then you don't need it anymore. I wrote a suite of tools that does all this and dumps the NFS transactions as JSON, maybe it can be useful: https://github.com/mprovost/NFStash https://github.com/mprovost/NFStash