3 ms·

I have some experience in running pretty large compute clusters with NFS as the way everything is accessed, so I feel your pain. We had an environment where all

I have some experience in running pretty large compute clusters with NFS as the way everything is accessed, so I feel your pain. We had an environment where all the binaries themselves were in NFS, as well as the data being processed by the jobs (on different servers at least… the ones hosting the binaries were the “tools” servers), and so we had thousands of machines hitting the tools NFS mounts thousands of times per second. A typical issue was jobs which ran a bash script which calls perl a bunch. Perl is on NFS. Oh and perl isn’t just on NFS, but the path to it is a symlink, which resolved to another symlink, which resolved to another symlink, which finally resolved to the perl binary. So that’s a half-dozen stat() calls per each call to `perl`. (None of these are cached, because metadata wasn’t cached, only the file contents were…) Said bash script would use ` | perl -pe` instead of grep, etc… Oh and typical perl programs called by these scripts had a bunch of modules they needed. Guess where the modules are? NFS! Not just NFS, but behind their own sets of symlinks (gotta be able to version your perl code! Why not use symlinks!) Some of these symlinks would trigger an automount. So now your bash script’s call to `perl -pe ‘s/foo/bar’` had to go mount something from another tools server. Yay. Needless to say we paid the storage vendor a lot for support.

dekhn 4y ago
Yeah, college students routinely bring down clusters by submitting large jobs with naive IO patterns. A common strategy at supercomputer centers is to not mount the home dir at all, and force jobs to stage every single bit they need into the work node's filesystem. We put tons of artifacts and modules on NFS but our filers are designed to scale. Once, I even had to evaluate a product (Gear 6) which was a read-only cache of another NFS server, just to offload the heavy reads caused by cluster scripts. With all that said, the best cluster I ever ran was diskless- all the worker nodes mounted / and /home from a master node, booting straight into a root NFS filesystem.
mcguire 4y ago
AIX 3.2.5 (IIRC) had a bug in NFS where you could corrupt a filesystem's lock with the result that any process that touched the filesystem in any way would block. IBM's automountd was single threaded. Someone's code (specifically, a systems programming class assignment) would lock the FS with their home directory. After a period of inactivity the automounter would try to unmount the directory, blocking. The machine is now nearly useless.
- dekhn 4y ago
  Oh, Netapp had a similar bug in ONTAP. If you had a nearly full (98%) filesystem, snapshots enabled, and somebody did an rm -rf <MYDIROFLOTSOFDATA>, the whole filer would stop responding to all users for half an hour or more. Eventually we got a patch (custom for our enterprise) that limited the outage to anybody who had a current working directory that was the parent of the rm -rf'd dir. And we dropped NetApp shortly after.
com 4y ago
I hope I didn’t build that system. Sorry. The design is giving me flashbacks to the mid-90s in oil and gas.

dekhn 4y ago

Yeah, college students routinely bring down clusters by submitting large jobs with naive IO patterns. A common strategy at supercomputer centers is to not mount the home dir at all, and force jobs to stage every single bit they need into the work node's filesystem. We put tons of artifacts and modules on NFS but our filers are designed to scale. Once, I even had to evaluate a product (Gear 6) which was a read-only cache of another NFS server, just to offload the heavy reads caused by cluster scripts. With all that said, the best cluster I ever ran was diskless- all the worker nodes mounted / and /home from a master node, booting straight into a root NFS filesystem.

mcguire 4y ago

AIX 3.2.5 (IIRC) had a bug in NFS where you could corrupt a filesystem's lock with the result that any process that touched the filesystem in any way would block. IBM's automountd was single threaded. Someone's code (specifically, a systems programming class assignment) would lock the FS with their home directory. After a period of inactivity the automounter would try to unmount the directory, blocking. The machine is now nearly useless.

dekhn 4y ago
Oh, Netapp had a similar bug in ONTAP. If you had a nearly full (98%) filesystem, snapshots enabled, and somebody did an rm -rf <MYDIROFLOTSOFDATA>, the whole filer would stop responding to all users for half an hour or more. Eventually we got a patch (custom for our enterprise) that limited the outage to anybody who had a current working directory that was the parent of the rm -rf'd dir. And we dropped NetApp shortly after.

com 4y ago

I hope I didn’t build that system. Sorry. The design is giving me flashbacks to the mid-90s in oil and gas.