4 ms·

NFS is my favorite example of how to do distributed computing incorrectly. Take a filesystem API that assumes fast, local access, and that individual I/O operat

NFS is my favorite example of how to do distributed computing incorrectly. Take a filesystem API that assumes fast, local access, and that individual I/O operations are cheap and immediate (like stat()’ing a file or resolving a symlink), and suddenly make it so all these operations may have to block for hundreds/thousands of milliseconds while accessing something over a remote server potentially on the other side of the world. It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… NFS locking up is the most common answer to the typical interview question “what could cause a load average in the hundreds on a machine with zero CPU usage?”: because there’s a stuck NFS server and the pending IO operations put the processes in the run queue. I’m glad there’s a paper that articulates my thoughts about this subject better than I can.

zozbot234 4y ago
> It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… These things have nothing to do with distributed systems, they can happen on any machine. That's what the "Not ready reading drive A. Abort, Retry, Fail?_" prompt is for.
- dekhn 4y ago
  umount -l -f will unmount (logically) a stuck mount and let you mount something else there.
  - DonHopkins 4y ago
    And before that there was /etc/fastboot or the L1-A two finger salute!
pmontra 4y ago
I hit that a couple of weeks ago after some 30+ years. I forgot how it was. And the per client per directory file entry cache. It doesn't play nice with systems with producers that create files and consumers that look for them. By the way, Amazon's EFS is based on NFS.
_a_a_a_ 4y ago
Ok, so be constructive: what should an interface for remote file access look like?
- dekhn 4y ago
  Open(path) -> handle, Read(handle, offset, size), Write(handle, size), SnapshotForAppend(handle, newpath), Close And write is write-once, from the start of the file.
- ninkendo 4y ago
  The most important thing is that it should be asynchronous, and only asynchronous. A syscall like read() blocks by default (suspending the whole thread) until an NFS server comes back with a response. A more appropriate interface should not allow callers to operate in a blocking way. Blocking incurs a lot of problems, because the kernel ends up waiting a ridiculously long default timeout for the server to respond, and the calling thread is blocked the whole time. Making a remote-access API asynchronous (callback-based), or at least having O_NONBLOCK being the only allowed option, would help a lot here. But really, the other problem is that filesystems have too much baggage around assumptions of speed/locality/reliability that just blindly treating a remote server as if it’s a local filesystem is just a bad idea in the first place. Too much client software is written with the assumption that filesystems are fast (ie. you don’t need to cache it, stat()’s are cheap, etc) that it’s just better to not conflate the two.
dekhn 4y ago
GFS at Google was created for this very reason. In the early days, lots of disk servers exported NFS and lots of hosts mounted lots of NFS. This caused things like index building to break. What's interesting about GFS is that it's a purely RPC protocol with no kernel and the access library is linked into the application. That allowed a lot of freedom for both the server developers and the clients. It also forced the client developers to explicitly acknowledge they were working with a remote filesystem. (BTW, I work for an enterprise that uses NFS; we have 20PB in there and lots of HPC jobs and interactive users pounding on it. But we spent $$$ for a very good NFS server)
- ninkendo 4y ago
  I have some experience in running pretty large compute clusters with NFS as the way everything is accessed, so I feel your pain. We had an environment where all the binaries themselves were in NFS, as well as the data being processed by the jobs (on different servers at least… the ones hosting the binaries were the “tools” servers), and so we had thousands of machines hitting the tools NFS mounts thousands of times per second. A typical issue was jobs which ran a bash script which calls perl a bunch. Perl is on NFS. Oh and perl isn’t just on NFS, but the path to it is a symlink, which resolved to another symlink, which resolved to another symlink, which finally resolved to the perl binary. So that’s a half-dozen stat() calls per each call to `perl`. (None of these are cached, because metadata wasn’t cached, only the file contents were…) Said bash script would use ` | perl -pe` instead of grep, etc… Oh and typical perl programs called by these scripts had a bunch of modules they needed. Guess where the modules are? NFS! Not just NFS, but behind their own sets of symlinks (gotta be able to version your perl code! Why not use symlinks!) Some of these symlinks would trigger an automount. So now your bash script’s call to `perl -pe ‘s/foo/bar’` had to go mount something from another tools server. Yay. Needless to say we paid the storage vendor a lot for support.
  - dekhn 4y ago
    Yeah, college students routinely bring down clusters by submitting large jobs with naive IO patterns. A common strategy at supercomputer centers is to not mount the home dir at all, and force jobs to stage every single bit they need into the work node's filesystem. We put tons of artifacts and modules on NFS but our filers are designed to scale. Once, I even had to evaluate a product (Gear 6) which was a read-only cache of another NFS server, just to offload the heavy reads caused by cluster scripts. With all that said, the best cluster I ever ran was diskless- all the worker nodes mounted / and /home from a master node, booting straight into a root NFS filesystem.
nicolaslem 4y ago
Was this design issue addressed in the subsequent revisions of the protocol?
- ninkendo 4y ago
  TBH I’m not sure. The run queue (high load) issue seems the kind of thing that could be an issue with the client kernel, not necessarily the protocol, although I’m not sure of the details. It’s possible it’s fixed now, I haven’t been an NFS admin in 10 years :) But the real point is that the filesystem, as an abstraction, has too much baggage from clients that make local-first assumptions, like that stat()’s are cheap (let’s use lots of symlinks! Hey, why is the NFS server CPU usage so high from stat() calls?), or that the filesystem can be used for easy cache (which on NFS is kind of the opposite of what you want), etc… silently and transparently making your filesystem an NFS mount is a bad idea unless the software using that mount is aware that the other end is a (potentially slow) NFS server and not local disk. The two really shouldn’t be conflated.
  - KMag 4y ago
    > The run queue (high load) issue seems the kind of thing that could be an issue with the client kernel Under Linux, a high load roughly tells you that the system is bottlenecked on something, where that something isn't necessarily CPU cycles. If your system is bottlenecked on pending NFS requests, the load will be high, and that's not the result of any kernel shortcomings (unless one blames blocking file I/O on the difficulty of correct use of exposed async/non-blocking file I/O syscalls). The high load average is a matter of accounting[0]. Under Linux, tasks (threads) in I/O wait are considered runnable, although they're not scheduled for CPU time. Other OSes (e.g. most Unix descendants, probably WinNT) make other accounting decisions, but either way, tasks blocked on I/O don't affect CPU-bound tasks. The Linux manner of load accounting isn't what I would have implemented at first, but if you're going to collapse system load into a single metric, then it seems best to include blocked tasks. If you've got high load, then it's time to dig deeper into more nuanced metrics anyway, so in retrospect, I agree with Linux's more holistic load accounting. [0] https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_lo...
DonHopkins 4y ago
Oh yeah, don't get me started about NFS! (Oops, too late.) I'll just link with short summaries: https://news.ycombinator.com/item?id=20007875 https://news.ycombinator.com/item?id=20007875 >There used to be a bug in the GatorBox Mac Localtalk-to-Ethernet NFS bridge that could somehow trick Unix into putting slashes into file names via NFS, which appeared to work fine, but then down the line Unix "restore" would totally shit itself. And here's another cool party trick you can baffle people with (not related to NFS, just an obscure MacOS quirk): >I just tried to create a file name on the Mac in Finder with a slash in it, and it actually let me! But Emacs dired says it actually ended up with a ":" in it. So then I tried to create a file name with a colon in it, and Finder said: "Try using a name with fewer characters or with no punctuation marks." Must be backwards compatibility for all those old Mac files with slashes in their name. Go figure! https://news.ycombinator.com/item?id=31820504 https://news.ycombinator.com/item?id=31820504 >NFS originally stood for "No File Security". >The NFS protocol wasn't just stateless, but also securityless! >Stewart, remember the open secret that almost everybody at Sun knew about, in which you could tftp a host's /etc/exports (because tftp was set up by default in a way that left it wide open to anyone from anywhere reading files in /etc) to learn the name of all the servers a host allowed to mount its file system, and then in a root shell simply go "hostname foo ; mount remote:/dir /mnt ; hostname `hostname`" to temporarily change the CLIENT's hostname to the name of a host that the SERVER allowed to mount the directory, then mount it (claiming to be an allowed client), then switch it back? >That's right, the server didn't bother checking the client's IP address against the host name it claimed to be in the NFS mountd request. That's right: the protocol itself let the client tell the server what its host name was, and the server implementation didn't check that against the client's ip address. Nice professional protocol design and implementation, huh? >Yes, that actually worked, because the NFS protocol laughably trusted the CLIENT to identify its host name for security purposes. That level of "trust" was built into the original NFS protocol and implementation from day one, by the geniuses at Sun who originally designed it. The network is the computer is insecure, indeed. >And most engineers at Sun knew that (and many often took advantage of it). NFS security was a running joke, thus the moniker "No File Security". But Sun proudly shipped it to customers anyway, configured with terribly insecure defaults that let anybody on the internet mount your file system. (That "feature" was undocumented, of course.) https://news.ycombinator.com/item?id=31820891 https://news.ycombinator.com/item?id=31820891 >Stewart, I think "sucks" is a pretty fair description of a protocol that actually trusted the client to tell the server what its host name is, before the server checked that the host name appears in /etc/exports, without verifying the client's ip address. On a system that makes /etc/exports easily publicly readable via tftp by default. https://news.ycombinator.com/item?id=31822138 https://news.ycombinator.com/item?id=31822138 >Speaking of YP (which I always thought sounded like a brand of moist baby poop towelettes), BSD, wildcard groups, SunRPC, and Sun's ingenuous networking and security and remote procedure call infrastructure, who remembers Jordan Hubbard's infamous rwall incident on March 31, 1987? https://news.ycombinator.com/item?id=31821646 https://news.ycombinator.com/item?id=31821646 >Another reason that NFS sucks: Anyone remember the Gator Box? It enabled you to trick NFS into putting slashes into the names of files and directories, which seemed to work at the time, but came back to totally fuck you later when you tried to restore a dump of your file system.

zozbot234 4y ago

> It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… These things have nothing to do with distributed systems, they can happen on any machine. That's what the "Not ready reading drive A. Abort, Retry, Fail?_" prompt is for.

dekhn 4y ago
umount -l -f will unmount (logically) a stuck mount and let you mount something else there.
- DonHopkins 4y ago
  And before that there was /etc/fastboot or the L1-A two finger salute!

pmontra 4y ago

I hit that a couple of weeks ago after some 30+ years. I forgot how it was. And the per client per directory file entry cache. It doesn't play nice with systems with producers that create files and consumers that look for them. By the way, Amazon's EFS is based on NFS.

_a_a_a_ 4y ago

Ok, so be constructive: what should an interface for remote file access look like?

dekhn 4y ago
Open(path) -> handle, Read(handle, offset, size), Write(handle, size), SnapshotForAppend(handle, newpath), Close And write is write-once, from the start of the file.
ninkendo 4y ago
The most important thing is that it should be asynchronous, and only asynchronous. A syscall like read() blocks by default (suspending the whole thread) until an NFS server comes back with a response. A more appropriate interface should not allow callers to operate in a blocking way. Blocking incurs a lot of problems, because the kernel ends up waiting a ridiculously long default timeout for the server to respond, and the calling thread is blocked the whole time. Making a remote-access API asynchronous (callback-based), or at least having O_NONBLOCK being the only allowed option, would help a lot here. But really, the other problem is that filesystems have too much baggage around assumptions of speed/locality/reliability that just blindly treating a remote server as if it’s a local filesystem is just a bad idea in the first place. Too much client software is written with the assumption that filesystems are fast (ie. you don’t need to cache it, stat()’s are cheap, etc) that it’s just better to not conflate the two.

dekhn 4y ago

GFS at Google was created for this very reason. In the early days, lots of disk servers exported NFS and lots of hosts mounted lots of NFS. This caused things like index building to break. What's interesting about GFS is that it's a purely RPC protocol with no kernel and the access library is linked into the application. That allowed a lot of freedom for both the server developers and the clients. It also forced the client developers to explicitly acknowledge they were working with a remote filesystem. (BTW, I work for an enterprise that uses NFS; we have 20PB in there and lots of HPC jobs and interactive users pounding on it. But we spent $$$ for a very good NFS server)

ninkendo 4y ago
I have some experience in running pretty large compute clusters with NFS as the way everything is accessed, so I feel your pain. We had an environment where all the binaries themselves were in NFS, as well as the data being processed by the jobs (on different servers at least… the ones hosting the binaries were the “tools” servers), and so we had thousands of machines hitting the tools NFS mounts thousands of times per second. A typical issue was jobs which ran a bash script which calls perl a bunch. Perl is on NFS. Oh and perl isn’t just on NFS, but the path to it is a symlink, which resolved to another symlink, which resolved to another symlink, which finally resolved to the perl binary. So that’s a half-dozen stat() calls per each call to `perl`. (None of these are cached, because metadata wasn’t cached, only the file contents were…) Said bash script would use ` | perl -pe` instead of grep, etc… Oh and typical perl programs called by these scripts had a bunch of modules they needed. Guess where the modules are? NFS! Not just NFS, but behind their own sets of symlinks (gotta be able to version your perl code! Why not use symlinks!) Some of these symlinks would trigger an automount. So now your bash script’s call to `perl -pe ‘s/foo/bar’` had to go mount something from another tools server. Yay. Needless to say we paid the storage vendor a lot for support.
- dekhn 4y ago
  Yeah, college students routinely bring down clusters by submitting large jobs with naive IO patterns. A common strategy at supercomputer centers is to not mount the home dir at all, and force jobs to stage every single bit they need into the work node's filesystem. We put tons of artifacts and modules on NFS but our filers are designed to scale. Once, I even had to evaluate a product (Gear 6) which was a read-only cache of another NFS server, just to offload the heavy reads caused by cluster scripts. With all that said, the best cluster I ever ran was diskless- all the worker nodes mounted / and /home from a master node, booting straight into a root NFS filesystem.

nicolaslem 4y ago

Was this design issue addressed in the subsequent revisions of the protocol?

ninkendo 4y ago
TBH I’m not sure. The run queue (high load) issue seems the kind of thing that could be an issue with the client kernel, not necessarily the protocol, although I’m not sure of the details. It’s possible it’s fixed now, I haven’t been an NFS admin in 10 years :) But the real point is that the filesystem, as an abstraction, has too much baggage from clients that make local-first assumptions, like that stat()’s are cheap (let’s use lots of symlinks! Hey, why is the NFS server CPU usage so high from stat() calls?), or that the filesystem can be used for easy cache (which on NFS is kind of the opposite of what you want), etc… silently and transparently making your filesystem an NFS mount is a bad idea unless the software using that mount is aware that the other end is a (potentially slow) NFS server and not local disk. The two really shouldn’t be conflated.
- KMag 4y ago
  > The run queue (high load) issue seems the kind of thing that could be an issue with the client kernel Under Linux, a high load roughly tells you that the system is bottlenecked on something, where that something isn't necessarily CPU cycles. If your system is bottlenecked on pending NFS requests, the load will be high, and that's not the result of any kernel shortcomings (unless one blames blocking file I/O on the difficulty of correct use of exposed async/non-blocking file I/O syscalls). The high load average is a matter of accounting[0]. Under Linux, tasks (threads) in I/O wait are considered runnable, although they're not scheduled for CPU time. Other OSes (e.g. most Unix descendants, probably WinNT) make other accounting decisions, but either way, tasks blocked on I/O don't affect CPU-bound tasks. The Linux manner of load accounting isn't what I would have implemented at first, but if you're going to collapse system load into a single metric, then it seems best to include blocked tasks. If you've got high load, then it's time to dig deeper into more nuanced metrics anyway, so in retrospect, I agree with Linux's more holistic load accounting. [0] https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_lo...

DonHopkins 4y ago

Oh yeah, don't get me started about NFS! (Oops, too late.) I'll just link with short summaries: https://news.ycombinator.com/item?id=20007875 https://news.ycombinator.com/item?id=20007875 >There used to be a bug in the GatorBox Mac Localtalk-to-Ethernet NFS bridge that could somehow trick Unix into putting slashes into file names via NFS, which appeared to work fine, but then down the line Unix "restore" would totally shit itself. And here's another cool party trick you can baffle people with (not related to NFS, just an obscure MacOS quirk): >I just tried to create a file name on the Mac in Finder with a slash in it, and it actually let me! But Emacs dired says it actually ended up with a ":" in it. So then I tried to create a file name with a colon in it, and Finder said: "Try using a name with fewer characters or with no punctuation marks." Must be backwards compatibility for all those old Mac files with slashes in their name. Go figure! https://news.ycombinator.com/item?id=31820504 https://news.ycombinator.com/item?id=31820504 >NFS originally stood for "No File Security". >The NFS protocol wasn't just stateless, but also securityless! >Stewart, remember the open secret that almost everybody at Sun knew about, in which you could tftp a host's /etc/exports (because tftp was set up by default in a way that left it wide open to anyone from anywhere reading files in /etc) to learn the name of all the servers a host allowed to mount its file system, and then in a root shell simply go "hostname foo ; mount remote:/dir /mnt ; hostname `hostname`" to temporarily change the CLIENT's hostname to the name of a host that the SERVER allowed to mount the directory, then mount it (claiming to be an allowed client), then switch it back? >That's right, the server didn't bother checking the client's IP address against the host name it claimed to be in the NFS mountd request. That's right: the protocol itself let the client tell the server what its host name was, and the server implementation didn't check that against the client's ip address. Nice professional protocol design and implementation, huh? >Yes, that actually worked, because the NFS protocol laughably trusted the CLIENT to identify its host name for security purposes. That level of "trust" was built into the original NFS protocol and implementation from day one, by the geniuses at Sun who originally designed it. The network is the computer is insecure, indeed. >And most engineers at Sun knew that (and many often took advantage of it). NFS security was a running joke, thus the moniker "No File Security". But Sun proudly shipped it to customers anyway, configured with terribly insecure defaults that let anybody on the internet mount your file system. (That "feature" was undocumented, of course.) https://news.ycombinator.com/item?id=31820891 https://news.ycombinator.com/item?id=31820891 >Stewart, I think "sucks" is a pretty fair description of a protocol that actually trusted the client to tell the server what its host name is, before the server checked that the host name appears in /etc/exports, without verifying the client's ip address. On a system that makes /etc/exports easily publicly readable via tftp by default. https://news.ycombinator.com/item?id=31822138 https://news.ycombinator.com/item?id=31822138 >Speaking of YP (which I always thought sounded like a brand of moist baby poop towelettes), BSD, wildcard groups, SunRPC, and Sun's ingenuous networking and security and remote procedure call infrastructure, who remembers Jordan Hubbard's infamous rwall incident on March 31, 1987? https://news.ycombinator.com/item?id=31821646 https://news.ycombinator.com/item?id=31821646 >Another reason that NFS sucks: Anyone remember the Gator Box? It enabled you to trick NFS into putting slashes into the names of files and directories, which seemed to work at the time, but came back to totally fuck you later when you tried to restore a dump of your file system.