4 ms·

> stale file handles... This is still a problem and I’m not sure how to fix it. I don’t understand how real NFS servers do this, maybe they just have a really b

> stale file handles... This is still a problem and I’m not sure how to fix it. I don’t understand how real NFS servers do this, maybe they just have a really big cache? While some implementations probably cache things, most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) NFS servers can then just expose this along with some internal stuff (like export id for internal permissions checks etc) This can lead to some interesting misfeature if a user can guess the handles -- on most filesystems it'll just be something like the inode number and some extra stuff that depending on the filesystem can be contiguously allocated, so +1 can work -- in that open by handle at won't check that the user has access to the full path: so a malign user could access some/subdir/file if file itself is open even if subdir isn't accessible. Thanksfully most recent filesystems allocate inodes pseudo-randomly, so this won't work well for most people. (In this particular case, you could have commit id + a big hash table with paths inside the repo? Most recently large repos would fit in memory for one single revision and layout doesn't change all that much between commits, so that could work fairly well a bit past linux kernel size. For humongous repos some more tricks might be needed though)

EdSchouten 3y ago
Worth mentioning that if you use NFSv4, file handles can be up to 128 bytes in size. That’s big enough that you can sometimes get away with dumping the entire file state in it. For Buildbarn (a distributed build cluster for Bazel) I also wrote an NFSv4 server in Go to act as a lazy-loading file system for input files. As the remote execution protocol uses Content Addressable Storage, I ended up dumping the entire SHA-256, size and permission bits of files in the NFSv4 file handle. This allows me to reconstruct files as needed. Some notes I made at the time (which are written with Buildbarn knowledge in mind): https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.md https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....
cesarb 3y ago
> most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) On Windows, access to this is controlled by the confusingly named "Bypass traverse checking" (aka SeChangeNotifyPrivilege) permission (see for instance https://techcommunity.microsoft.com/t5/windows-blog-archive/the-bypass-traverse-checking-or-is-it-the-change-notify/ba-p/723428 https://techcommunity.microsoft.com/t5/windows-blog-archive/... for some information about it; I recall once reading an article from either Raymond Chen or Larry Osterman explaining the naming of that permission, but can't find it at the moment).

EdSchouten 3y ago

Worth mentioning that if you use NFSv4, file handles can be up to 128 bytes in size. That’s big enough that you can sometimes get away with dumping the entire file state in it. For Buildbarn (a distributed build cluster for Bazel) I also wrote an NFSv4 server in Go to act as a lazy-loading file system for input files. As the remote execution protocol uses Content Addressable Storage, I ended up dumping the entire SHA-256, size and permission bits of files in the NFSv4 file handle. This allows me to reconstruct files as needed. Some notes I made at the time (which are written with Buildbarn knowledge in mind): https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.md https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....

cesarb 3y ago

> most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) On Windows, access to this is controlled by the confusingly named "Bypass traverse checking" (aka SeChangeNotifyPrivilege) permission (see for instance https://techcommunity.microsoft.com/t5/windows-blog-archive/the-bypass-traverse-checking-or-is-it-the-change-notify/ba-p/723428 https://techcommunity.microsoft.com/t5/windows-blog-archive/... for some information about it; I recall once reading an article from either Raymond Chen or Larry Osterman explaining the naming of that permission, but can't find it at the moment).