11 ms·

NFS: The Early Years

176 points by chmaynard 4y ago

hsbauauvhabzb 4y ago
I’ve never really understood how permissions are supposed to work with NFS - are user IDs supposed to be consistent across systems? Is there a sane easy way to implement authentication? Last Time I tried iirc my options were LDAP or nil.
- pinewurst 4y ago
  Yes, administered by YP/NIS in the Sun era.
- chasil 4y ago
  Under NFSv2 and NFSv3, the numeric user and group id is used to determine permission, and these must be aligned between the client and server. I have an oracle uid 60 on an older system that maps as elcaro on an NFS client (because I have a different oracle user there as uid 54321). Under NFSv4, direct uid/gid is no longer used, but the RPC.idmapd process determines privilege. I'm not really sure how it works beyond continuing to work when uid/gid synchronization is in place for NFSv3 and the connection is upgraded. There is also an NFS ACL standard, but I don't know anything about it.
  - jjoonathan 4y ago
    > these must be aligned between the client and server looooool (Seriously, though, could someone tell me why this was supposed to make sense?)
    - toast0 4y ago
      If you've got centralized account management, it can work. Sending a fixed length 16-bit numeric id rather than a variable width username is a lot easier. I've worked somewhere with a lot of NFS, and they had centralized account management, so everything was fine other than actual security, at least until we hit the limit of 16-bit uids. That place had a different centralized account management for production, so uids weren't consistent between corp and prod, but NFS in prod was very limited. (And you wouldn't nfs between corp and prod either) I worked somewhere else without real centralized management of accounts on prod, and it was a PITA to bring that back under control, when it started becoming important. Even without intentional use of uids, it's convenient that they all line up on all servers; and it's a pain to change a uid that already exists on the system.
    - neilv 4y ago
      At the time, Sun NFS clients would receive equivalents of `/etc/passwd` over the network, using the YP service (later renamed NIS). Like much of Unix, it was worse-is-better, and pretty productive for a site. (Well, until there was a problem reaching the NFS server, or until there was a problem with an application license manager that everyone needed.)
    - acdha 4y ago
      > (Seriously, though, could someone tell me why this was supposed to make sense?) Think about the environment it was originally used in — large organizations, computers which cost as much as a car, LANs which aren't easily accessible (e.g. the Unix people have access but laptops are expensive oddity and the sales people are probably sitting in front of a DOS box or shelled into that Unix server), etc. It's more defensible when your unix administrator is going to configure each of the servers to use the same NIS user directory. All of that broke down when IP networking became the default, every desk in the building had a network port, and things like WiFi and laptops completely blew away the idea that the clients were managed by a single administrative group.
  - zaarn 4y ago
    the NFSv4 ACL Standard is tangentially related to NFS, the TL;DR is that it replicates the kind of ACLs you can create under Windows (ie, seperate "Write to File" and "Append to File" into different permission bits, make inheritance configurable, etc.). The TrueNAS people (ixsystems) have a patch to bring it to Linux and ZFS; though from what I've heard upstream LKML lists aren't too enthused since they'd rather see this being used by an in-kernel filesystem.
    - trasz 4y ago
      The ZFS support is there since the beginning (and used in Solaris and FreeBSD); Linux glue code is missing though.
      - zaarn 4y ago
        As mentioned, there is working glue code in form of the TrueNAS patches, which you can apply yourself if you need to. The Solaris support for NFS v4 ACLs is in my experience... subpar. It doesn't really integrate with any of the tooling (not that Solaris tooling is great to begin with).
        trasz 4y ago
        Hopefully the Linux glue code doesn’t use the richacls abomination?
        zaarn 4y ago
        It doesn't to my knowledge. It works out of the box and integrates with Samba.
    - jabl 4y ago
      For Linux there's the richacl's (https://en.wikipedia.org/wiki/Richacls https://en.wikipedia.org/wiki/Richacls ) which was an attempt to add NFSv4/Windows style ACL's to the Linux VFS. It never went upstream though, AFAICT largely because the VFS maintainer thought such ACL's are stupid.
  - geertj 4y ago
    > Under NFSv2 and NFSv3, the numeric user and group id is used to determine permission, and these must be aligned between the client and server. Technically the server doesn't need to have a UID/GID database that's aligned with the client, what's required is that all clients of the same server are aligned. The server will take the numerical UID/GIDs from the RPC sent by the client and perform Posix style permission checks using the owner UID, owner GID, and mode bits stored in the inode of the file or directory. The server doesn't need to known what user the UID corresponds to.
    - smarks 4y ago
      Right. At least at Sun through the 1990s, when everybody had their own workstations, many network nodes had local filesystems, so they were both NFS clients and NFS servers. For this to work well it pretty much required that UIDs/GIDs were globally consistent. This was maintained using YP/NIS. But Sun was too big for a single YP/NIS domain, so there was a hack where each YP/NIS master was populated via some kind of uber-master database. At least at one point, this consisted of plain text files on a filesystem that was NFS-mounted by every YP/NIS master.... This was all terribly insecure. Since everybody had root on their own workstations, you could `su root` and then `su somebody` to get processes running with their UID, and then you could read and write all their files over NFS. But remember, this was back in the day when we sent passwords around in the clear, we used insecure tools like telnet and ftp and BSD tools like rsh/rcp/rlogin. So NFS was "no more insecure" than anything else running on the network. But that was ok, because everything was behind a firewall. (Some sarcasm in those last bits, in case it wasn't obvious.)
      - KateLawson 4y ago
        Sun did have a firewall by the early 90's. It had application-level proxies, and you'd have to configure applications to bounce through it if you wanted to get to the Internet. In many ways, this was more secure than today's default for firewalls where you can make any outbound connection you want but only the inbound connections are filtered. Note that I'm not arguing that Sun was a leader in security, but they did make some efforts that other companies didn't.
B1FF_PSUVM 4y ago
I can't believe they didn't toss in the "Not Working Filesystem" joke as a freebie. Another way to have thumb-twiddling fun was for all the machines in a building to reboot - power failure, whatever - and then spend a lot of time waiting for each other's cross-mounted NFS shares to come up. P.S. > such as AFS (the Andrew File System) That one prompted a senior engineer, on learning that it was to be deployed to undergrad's lab machines, to let slip "Why inflict it on the hapless livestock?"
- eddyfromtheblok 4y ago
  AFS had (has?) a nice ACL implementation, compared to the opportunity that NFS provided of mangling security with UID mismatches.
  - B1FF_PSUVM 4y ago
    I wasn't directly involved, but I think the AFS design was OK, the problem was that they were deploying "diskless" workstations (just the OS, no data on the local HD). Back in the day of 1 MHz machines, with 3 Mbps net, having your home dir on a network file system was a tad hopeful - or hopeless ...
incompatible 4y ago
I remember an issue that it's security model involved trusting the client. If you exported a file system to a PC, somebody could reboot the PC with Linux to get root and ignore the user permissions.
- bubblethink 4y ago
  I believe this can be fixed with kerberos, but in practice very few people probably deploy it that way.
- chasil 4y ago
  They can get every user except root (assuming that root squash is in place). NFSv3 and below trusts any uid/gids presented by the client unless they are squashed.
smarks 4y ago
Of course, the first commenter "willy" repeats the canard that statelessness makes no sense: > The very notion of a stateless filesystem is ridiculous. Filesystems exist to store state. It's the protocol that's stateless, not the filesystem. I thought the article made a reasonable attempt to explain that. Overall the article is reasonable but it omits one of the big issues with NFSv2, which is synchronous writes. Those Sun NFS implementations were based on Sun's RPC system; the server was required not to reply until the write had been committed to stable storage. There was a mount option to disable this, but if you enabled it, it exposed you to data corruption. Certain vendors (SGI, if I recall correctly) at some point claimed their NFS was faster than Sun's, but it implemented asynchronous writes. This resulted in the expected arguments over protocol compliance and reliability vs. performance. This phenomenon led to various hardware "NFS accelerator" solutions that put an NVRAM write cache in front of the disk in order to speed up synchronous writes. I believe Legato and the still-existing NetApp were based on such technology. Eventually the synchronous writes issue was resolved, possibly by NFSv3, though the details escape me.
- wahern 4y ago
  NFS is basically the original S3. Both are useful for similar scenarios (maybe a slightly narrow subset for NFS (especially in later incarnations), and the semantics of both break down in similar ways. I've always just presumed the development of EFS recapitulated the evolution of NFS, in many cases quite literally, considering the EFS protocol is a flavor of NFS. S3 buckets are just blobs with GUIDs in a flat namespace, which is literally what stateless NFS is--every "file" has a persistent UID (GUID if you assume host identifiers are unique), providing a simple handle for submitting idempotent block-oriented read and write operations. Theoretically, EFS could just be a fairly simple interface over S3, especially if you can implicitly wave away many of the caveats (e.g. wrt shared writes) by simply pointing out they have existed and mostly been tolerated in NFS environments for decades.
  - geertj 4y ago
    (PM-T on the EFS team) S3 and EFS actually are quite different. Files on EFS are update-able, rename-able and link-able (I.e what’s expected from a file system), while S3 objects are immutable once they are created. This comes from the underlying data structures. EFS uses inodes and directories while S3 is more of a flat map. Protocol-wise EFS uses standard NFS 4.1. We added some optional innovations outside the protocol that you can use through our mount helper (mount.efs). This includes in-transit encryption with TLS (you can basically talk TLS to our endpoint and we will detect that automatically), and we support strong client auth using SigV4 over x509 client certificate.
mangix 4y ago
Isn't SMB better than NFS?
- candiddevmike 4y ago
  Depends on the use case. SMB auth is more robust and easier to integrate with AD, but NFS is simpler and typically faster for file access and transfer speeds. SMB is good for shares used by end users, NFS is good for shares used by services.
- heavyset_go 4y ago
  I've found NFSv4 to be more stable and performant than SMB when using it between Linux machines. Seems to handle multiple concurrent clients well, too.
  - kevin_thibedeau 4y ago
    NFS works great until you lose the network and your client locks up.
    - heavyset_go 4y ago
      Mount your NFS shares with the `intr` and `vers=4.2` options.
    - teddyh 4y ago
      Are you really talking about NFSv4?
- colordrops 4y ago
  It at least doesn't lock anything up that has a file open when the network goes down. NFS is a nightmare with that. NFS is more idiomatic on *nix but still a huge pain when dealing with matching file perms across systems.
  - dikei 4y ago
    > It at least doesn't lock anything up that has a file open when the network goes down. NFS is a nightmare with that. Yeah, we've been bitten by this too, around once a year, even with our fairly reliable and redundant network. It's a PITA, your process just hang and there's no way to even kill it except restarting the server.
pram 4y ago
My least favorite NFS thing I've dealt with was the integrated client in Oracle. Yes, really. Seemed almost impossible to figure out anything when stuff was broken. https://docs.oracle.com/en/database/oracle/oracle-database/19/ssdbi/about-direct-nfs-client-mounts-to-nfs-storage-devices.html#GUID-31591084-74BD-4B66-8C5B-68BF0FEE8750 https://docs.oracle.com/en/database/oracle/oracle-database/1...
- brazzledazzle 4y ago
  I had no idea this existed. This has to be considered masochism in software form.
netfortius 4y ago
Has AWS fixed the mounting of EFS under Windows based EC2s?
- geertj 4y ago
  We currently don't support Windows as a client. The main reason is that the builtin Windows NFS client is v3, while EFS is v4+. Some people have reported success mounting EFS on Windows using a third party NFS client, but I cannot comment on how well this works.
js2 4y ago
> One way is to generate a string that will be unique across all clients — possibly with host name, process ID, and timestamp — and then create a temporary file with this string as both name and content. This file is then (hard) linked to the name for the lock file. If the hard-link succeeds, the lock has been claimed. If it fails because the name already exists, then the application can read the content of that file. If it matches the generated unique string, then the error was due to a retransmit and again the lock has been claimed. Otherwise the application needs to sleep and try again. I seem to recall checking the link count on the temporary file is all that was needed: https://github.com/jaysoffian/dotlock https://github.com/jaysoffian/dotlock (That code is from 2001 or so.)
mprovost 4y ago
One of the reasons that NFS was so popular is that Sun gave away the protocol implementation. It's really interesting to see the technology behind that though. The whole protocol (both versions 2 and 3) is defined in a single file.[0] You run that file through the rpcgen program which spits out some .c and .h files which you can just link into your program. Of course on the server side you actually need to implement the code to handle those procedures, but on the client side it's all there, ready to use. [0] https://github.com/freebsd/freebsd-src/blob/main/include/rpcsvc/nfs_prot.x https://github.com/freebsd/freebsd-src/blob/main/include/rpc...
angrygoat 4y ago
A fun glitch with NFS with Linux serving out ext2/ext3 was getting -ENOENT when calling unlink() on a very large file (for the time.) The kernel on the server would do the work of unlinking the file which might take many seconds. In the meanwhile, the client would timeout, and make another NFS call to unlink. ext2/3 would have removed the path from the visible filesystem namespace, even though unlink hadn't complted, so this second call would return ENOENT. Somewhat confusing to users!
reacharavindh 4y ago
An unpopular opinion, but NFS is super handy and useful in reliable private networks with centralised authentication. Sure, it has its downsides that are being worked on with newer versions of the protocol(4+) with addition complexity, but it sure is useful in closely controlled setups like for HPC clusters. I ran a HPC cluster for an University, and relied upon good old NFSv3 for shared file storage(both home directories, and research datasets). In addition I also built out a big set of softwares compiled in one server and made available to the entire cluster via a read-only NFS mount point. The whole thing works so reliably without any hiccups whatsoever. To over some the limitations of authentication and authorisation with NFS storage, we use a centralised FreeIPA server that allows all machines in the cluster have the same UID/GID mapping everywhere. As a cream on top, the storage we expose over NFS is ZFS, that integrates nicely with NFS. Update 1: Yes, data security is a bit of an afterthought with NFS. As anybody in my network with physical access can mount my central storage to another server physically and access data as long as they can recreate UID/GID locally.. but, if I let someone to do this physically, I have bigger problems to deal with first.
- teddyh 4y ago
  You can fix the network access problem by using IPsec.
  - throw0101a 4y ago
    NFS over TLS: * https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-rpc-tls https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-rpc-t...
    - teddyh 4y ago
      Latest draft expired more than a year ago?
      - throw0101a 4y ago
        Implementations for Linux and FreeBSD: * https://patchwork.kernel.org/project/cifs-client/cover/165030062272.5246.16956092606399079004.stgit@oracle-102.nfsv4.dev/ https://patchwork.kernel.org/project/cifs-client/cover/16503... * https://www.freshports.org/sysutils/nfs-over-tls/ https://www.freshports.org/sysutils/nfs-over-tls/ Activity on the NFSv4 mailing list: * https://mailarchive.ietf.org/arch/browse/nfsv4/ https://mailarchive.ietf.org/arch/browse/nfsv4/ But no recent commits to the draft: * https://github.com/chucklever/i-d-rpc-tls https://github.com/chucklever/i-d-rpc-tls ¯\_(ツ)_/¯
DonHopkins 4y ago
Another reason that NFS sucks: Anyone remember the Gator Box? It enabled you to trick NFS into putting slashes into the names of files and directories, which seemed to work at the time, but came back to totally fuck you later when you tried to restore a dump of your file system. The NFS protocol itself didn't disallow slashes in file names, so the NFS server would accept them without question from any client, silently corrupting the file system without any warning. Thanks, NFS! Oh and here's a great party trick that will totally blow your mind: On a Mac, use the Finder to create a folder or file whose name is today's date, like "2022/06/21", or anything with a slash in it. Cool, huh? Bet you didn't think you could do that! Now open a shell and "ls -l" the directory containing the file you just created with slashes in it name. What just happened there? Now try creating a folder or file whose name is the current time, or anything with colons, like "10:35:43". Ha ha! Don't worry, it's totally harmless and won't trash your file system or backups like NFS with a Gator Box would. https://news.ycombinator.com/item?id=20007875 https://news.ycombinator.com/item?id=20007875 DonHopkins on May 25, 2019 | parent | context | favorite | on: Why Does Windows Really Use Backslash as Path Sepa... There used to be a bug in the GatorBox Mac Localtalk-to-Ethernet NFS bridge that could somehow trick Unix into putting slashes into file names via NFS, which appeared to work fine, but then down the line Unix "restore" would totally shit itself. That was because Macs at the time (1991 or so) allowed you to use slashes (and spaces of course, but not colons, which it used a a path separator), and of course those silly Mac people, being touchy feely humans instead of hard core nerds, would dare to name files with dates like "My Spreadsheet 01/02/1991". https://en.wikipedia.org/wiki/GatorBox https://en.wikipedia.org/wiki/GatorBox Unix-Haters Handbook https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.txt https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.tx... Don't Touch That Slash! UFS allows any character in a filename except for the slash (/) and the ASCII NUL character. (Some versions of Unix allow ASCII characters with the high-bit, bit 8, set. Others don't.) This feature is great — especially in versions of Unix based on Berkeley's Fast File System, which allows filenames longer than 14 characters. It means that you are free to construct informative, easy-to-understand filenames like these: 1992 Sales Report Personnel File: Verne, Jules rt005mfkbgkw0 . cp Unfortunately, the rest of Unix isn't as tolerant. Of the filenames shown above, only rt005mfkbgkw0.cp will work with the majority of Unix utili- ties (which generally can't tolerate spaces in filenames). However, don't fret: Unix will let you construct filenames that have control characters or graphics symbols in them. (Some versions will even let you build files that have no name at all.) This can be a great security feature — especially if you have control keys on your keyboard that other people don't have on theirs. That's right: you can literally create files with names that other people can't access. It sort of makes up for the lack of serious security access controls in the rest of Unix. Recall that Unix does place one hard-and-fast restriction on filenames: they may never, ever contain the magic slash character (/), since the Unix kernel uses the slash to denote subdirectories. To enforce this requirement, the Unix kernel simply will never let you create a filename that has a slash in it. (However, you can have a filename with the 0200 bit set, which does list on some versions of Unix as a slash character.) Never? Well, hardly ever. Date: Mon, 8 Jan 90 18:41:57 PST From: sun!wrs!yuba!steve@decwrl.dec.com (Steve Sekiguchi) Subject: Info-Mac Digest V8 #3 5 I've got a rather difficult problem here. We've got a Gator Box run- ning the NFS/AFP conversion. We use this to hook up Macs and Suns. With the Sun as a AppleShare File server. All of this works great! Now here is the problem, Macs are allowed to create files on the Sun/ Unix fileserver with a "/" in the filename. This is great until you try to restore one of these files from your "dump" tapes, "restore" core dumps when it runs into a file with a "/" in the filename. As far as I can tell the "dump" tape is fine. Does anyone have a suggestion for getting the files off the backup tape? Thanks in Advance, Steven Sekiguchi Wind River Systems sun!wrs!steve, steve@wrs.com Emeryville CA, 94608 Apparently Sun's circa 1990 NFS server (which runs inside the kernel) assumed that an NFS client would never, ever send a filename that had a slash inside it and thus didn't bother to check for the illegal character. We're surprised that the files got written to the dump tape at all. (Then again, perhaps they didn't. There's really no way to tell for sure, is there now?)
JBiserkov 4y ago
Need for Speed
Someone 4y ago
I don’t understand the “as both the client and server must maintain the same list of locks for each client” part of: “NLM (the Network Lock Manager). This allows the client to request a byte-range lock on a given file (identified using an NFS file handle), and allows the server to grant it (or not), either immediately or later. Naturally this is an explicitly stateful protocol, as both the client and server must maintain the same list of locks for each client.”* There is no “the client”, so if clients have to maintain that information, how is it distributed to all clients (including those that will make their first request in the future)? How does the server even know all clients, given the statelessness of the protocol? Or does that locking only work for requests from the same server? Or does the client keep that information only so that it can unlock the ranges when it discovers the process that locked the range exits/crashed? Is it even correct to assume such range locks can’t be created by another process than the one that will delete them (say after the first process forked)?
infologics 4y ago
[dead]
rwmj 4y ago
The thing I hated in NFSv2/3 (and fixed, finally, in NFSv4) is the use of random port numbers, making it difficult to have an NFS server with a firewall. Yes you can set 5(!) environment variables in an obscure file to fix the port numbers. NFSv4 routes everything over port 2049, but unfortunately breaks user mapping, I guess we can't have everything.
- jabl 4y ago
  In the current NFSv4 RFC (https://datatracker.ietf.org/doc/html/rfc7530#section-5.9 https://datatracker.ietf.org/doc/html/rfc7530#section-5.9 ) it says: > To provide a greater degree of compatibility with NFSv3, which identified users and groups by 32-bit unsigned user identifiers and group identifiers, owner and group strings that consist of ASCII- encoded decimal numeric values with no leading zeros can be given a special interpretation by clients and servers that choose to provide such support. The receiver may treat such a user or group string as representing the same user as would be represented by an NFSv3 uid or gid having the corresponding numeric value. I'm not sure how common this extension is, at least the Linux server and client support it out of the box. Isilon also supports it, but it must be explicitly enabled.
  - geertj 4y ago
    > I'm not sure how common this extension is, at least the Linux server and client support it out of the box. Isilon also supports it, but it must be explicitly enabled. It is very common. I’m not aware of a v4 server that does not support this. Originally v4 went all in on Kerberos (GSSAPI technically) to provide strong multi user auth. This is the reason that users and groups are represented as strings. This approach works reasonably well on Windows with SMB since you have AD there giving you Kerberos and a shared (LDAP) directory. AD is also deeply integrated in the OS for things like credential management and provisioning. The approach did not work so well on Linux where not everyone is running AD or even some kind of directory. This caused the protocol designers to make Kerberos optional in v4.1. I guess the spec authors already knew that Kerberos was going to be difficult because I just checked and the numerical-strong-user-as-posix-id workaround was already present in the original 4.0 spec.
irusensei 4y ago
Interesting fact: Linux kernel.org website used to offer public over the internet NFS (and SMB) mirrors. https://web.archive.org/web/19980130085039/http://www.kernel.org/ https://web.archive.org/web/19980130085039/http://www.kernel...
- chasil 4y ago
  Some of the big public FTP servers also did this. I think I remember it on one of the sunsite servers.
atulsnj 4y ago
Silly me, I was hoping for a game review.
atulsnj 4y ago
Silly me, I clicked hoping for a game review.
bluedino 4y ago
NFS can be...interesting to configure. Most of your enterprise storage systems support it (as well as the smaller stuff like Synology) and there's all kinds of 'but what about if we need to...' https://www.delltechnologies.com/asset/en-us/products/storage/industry-market/h17240_wp_isilon_onefs_nfs_design_considerations_bp.pdf https://www.delltechnologies.com/asset/en-us/products/storag...
smcameron 4y ago
I noticed that NFS became a lot less important to me around the time that distributed version control software (specifically CVS) became popular. Suddenly the main reason I was using NFS, to share our RCS source code repo, disappeared.
lizknope 4y ago
Did anybody use Auspex NFS servers in the late 1990's? My first job in 1997 was all Sun / SPARC based (semiconductor design) and we had a bunch of Auspex file servers. It looks like they went under in 2003 and NetApp became the dominant company.
- DonHopkins 4y ago
  Guy Harris worked at Sun from 1985 - 1988, then on NFS at Auspex from 1988 - 1994, then he worked at NetApp from 1994 - 2003. After that he worked at Apple, now he's been hacking Wireshark for more than 23 years, since 1998! https://ask.wireshark.org/users/12/guy-harris/ https://ask.wireshark.org/users/12/guy-harris/ https://www.wireshark.org/docs/wsug_html_chunked/ChIntroHistory.html https://www.wireshark.org/docs/wsug_html_chunked/ChIntroHist...
- acdha 4y ago
  We got a lot of value out of ours in an academic research context but switched around 2002 when they just couldn't manage to be price-competitive. We ended up with Spinnaker, Isilon, and NetApp boxes which were all Linux-based, although with some custom extensions (if memory serves, we found a bunch of correctness issues running fsx against Isilon because their filesystem implementation wasn't as robust as the stock Linux filesystems).

176 points by chmaynard 4y ago

I’ve never really understood how permissions are supposed to work with NFS - are user IDs supposed to be consistent across systems? Is there a sane easy way to implement authentication? Last Time I tried iirc my options were LDAP or nil.

pinewurst 4y ago
Yes, administered by YP/NIS in the Sun era.
chasil 4y ago
Under NFSv2 and NFSv3, the numeric user and group id is used to determine permission, and these must be aligned between the client and server. I have an oracle uid 60 on an older system that maps as elcaro on an NFS client (because I have a different oracle user there as uid 54321). Under NFSv4, direct uid/gid is no longer used, but the RPC.idmapd process determines privilege. I'm not really sure how it works beyond continuing to work when uid/gid synchronization is in place for NFSv3 and the connection is upgraded. There is also an NFS ACL standard, but I don't know anything about it.
- jjoonathan 4y ago
  > these must be aligned between the client and server looooool (Seriously, though, could someone tell me why this was supposed to make sense?)
  - toast0 4y ago
    If you've got centralized account management, it can work. Sending a fixed length 16-bit numeric id rather than a variable width username is a lot easier. I've worked somewhere with a lot of NFS, and they had centralized account management, so everything was fine other than actual security, at least until we hit the limit of 16-bit uids. That place had a different centralized account management for production, so uids weren't consistent between corp and prod, but NFS in prod was very limited. (And you wouldn't nfs between corp and prod either) I worked somewhere else without real centralized management of accounts on prod, and it was a PITA to bring that back under control, when it started becoming important. Even without intentional use of uids, it's convenient that they all line up on all servers; and it's a pain to change a uid that already exists on the system.
  - neilv 4y ago
    At the time, Sun NFS clients would receive equivalents of `/etc/passwd` over the network, using the YP service (later renamed NIS). Like much of Unix, it was worse-is-better, and pretty productive for a site. (Well, until there was a problem reaching the NFS server, or until there was a problem with an application license manager that everyone needed.)
  - acdha 4y ago
    > (Seriously, though, could someone tell me why this was supposed to make sense?) Think about the environment it was originally used in — large organizations, computers which cost as much as a car, LANs which aren't easily accessible (e.g. the Unix people have access but laptops are expensive oddity and the sales people are probably sitting in front of a DOS box or shelled into that Unix server), etc. It's more defensible when your unix administrator is going to configure each of the servers to use the same NIS user directory. All of that broke down when IP networking became the default, every desk in the building had a network port, and things like WiFi and laptops completely blew away the idea that the clients were managed by a single administrative group.
- zaarn 4y ago
  the NFSv4 ACL Standard is tangentially related to NFS, the TL;DR is that it replicates the kind of ACLs you can create under Windows (ie, seperate "Write to File" and "Append to File" into different permission bits, make inheritance configurable, etc.). The TrueNAS people (ixsystems) have a patch to bring it to Linux and ZFS; though from what I've heard upstream LKML lists aren't too enthused since they'd rather see this being used by an in-kernel filesystem.
  - trasz 4y ago
    The ZFS support is there since the beginning (and used in Solaris and FreeBSD); Linux glue code is missing though.
    - zaarn 4y ago
      As mentioned, there is working glue code in form of the TrueNAS patches, which you can apply yourself if you need to. The Solaris support for NFS v4 ACLs is in my experience... subpar. It doesn't really integrate with any of the tooling (not that Solaris tooling is great to begin with).
      - trasz 4y ago
        Hopefully the Linux glue code doesn’t use the richacls abomination?
        zaarn 4y ago
        It doesn't to my knowledge. It works out of the box and integrates with Samba.
  - jabl 4y ago
    For Linux there's the richacl's (https://en.wikipedia.org/wiki/Richacls https://en.wikipedia.org/wiki/Richacls ) which was an attempt to add NFSv4/Windows style ACL's to the Linux VFS. It never went upstream though, AFAICT largely because the VFS maintainer thought such ACL's are stupid.
- geertj 4y ago
  > Under NFSv2 and NFSv3, the numeric user and group id is used to determine permission, and these must be aligned between the client and server. Technically the server doesn't need to have a UID/GID database that's aligned with the client, what's required is that all clients of the same server are aligned. The server will take the numerical UID/GIDs from the RPC sent by the client and perform Posix style permission checks using the owner UID, owner GID, and mode bits stored in the inode of the file or directory. The server doesn't need to known what user the UID corresponds to.
  - smarks 4y ago
    Right. At least at Sun through the 1990s, when everybody had their own workstations, many network nodes had local filesystems, so they were both NFS clients and NFS servers. For this to work well it pretty much required that UIDs/GIDs were globally consistent. This was maintained using YP/NIS. But Sun was too big for a single YP/NIS domain, so there was a hack where each YP/NIS master was populated via some kind of uber-master database. At least at one point, this consisted of plain text files on a filesystem that was NFS-mounted by every YP/NIS master.... This was all terribly insecure. Since everybody had root on their own workstations, you could `su root` and then `su somebody` to get processes running with their UID, and then you could read and write all their files over NFS. But remember, this was back in the day when we sent passwords around in the clear, we used insecure tools like telnet and ftp and BSD tools like rsh/rcp/rlogin. So NFS was "no more insecure" than anything else running on the network. But that was ok, because everything was behind a firewall. (Some sarcasm in those last bits, in case it wasn't obvious.)
    - KateLawson 4y ago
      Sun did have a firewall by the early 90's. It had application-level proxies, and you'd have to configure applications to bounce through it if you wanted to get to the Internet. In many ways, this was more secure than today's default for firewalls where you can make any outbound connection you want but only the inbound connections are filtered. Note that I'm not arguing that Sun was a leader in security, but they did make some efforts that other companies didn't.

B1FF_PSUVM 4y ago

I can't believe they didn't toss in the "Not Working Filesystem" joke as a freebie. Another way to have thumb-twiddling fun was for all the machines in a building to reboot - power failure, whatever - and then spend a lot of time waiting for each other's cross-mounted NFS shares to come up. P.S. > such as AFS (the Andrew File System) That one prompted a senior engineer, on learning that it was to be deployed to undergrad's lab machines, to let slip "Why inflict it on the hapless livestock?"

eddyfromtheblok 4y ago
AFS had (has?) a nice ACL implementation, compared to the opportunity that NFS provided of mangling security with UID mismatches.
- B1FF_PSUVM 4y ago
  I wasn't directly involved, but I think the AFS design was OK, the problem was that they were deploying "diskless" workstations (just the OS, no data on the local HD). Back in the day of 1 MHz machines, with 3 Mbps net, having your home dir on a network file system was a tad hopeful - or hopeless ...

incompatible 4y ago

I remember an issue that it's security model involved trusting the client. If you exported a file system to a PC, somebody could reboot the PC with Linux to get root and ignore the user permissions.

bubblethink 4y ago
I believe this can be fixed with kerberos, but in practice very few people probably deploy it that way.
chasil 4y ago
They can get every user except root (assuming that root squash is in place). NFSv3 and below trusts any uid/gids presented by the client unless they are squashed.

smarks 4y ago

Of course, the first commenter "willy" repeats the canard that statelessness makes no sense: > The very notion of a stateless filesystem is ridiculous. Filesystems exist to store state. It's the protocol that's stateless, not the filesystem. I thought the article made a reasonable attempt to explain that. Overall the article is reasonable but it omits one of the big issues with NFSv2, which is synchronous writes. Those Sun NFS implementations were based on Sun's RPC system; the server was required not to reply until the write had been committed to stable storage. There was a mount option to disable this, but if you enabled it, it exposed you to data corruption. Certain vendors (SGI, if I recall correctly) at some point claimed their NFS was faster than Sun's, but it implemented asynchronous writes. This resulted in the expected arguments over protocol compliance and reliability vs. performance. This phenomenon led to various hardware "NFS accelerator" solutions that put an NVRAM write cache in front of the disk in order to speed up synchronous writes. I believe Legato and the still-existing NetApp were based on such technology. Eventually the synchronous writes issue was resolved, possibly by NFSv3, though the details escape me.

wahern 4y ago
NFS is basically the original S3. Both are useful for similar scenarios (maybe a slightly narrow subset for NFS (especially in later incarnations), and the semantics of both break down in similar ways. I've always just presumed the development of EFS recapitulated the evolution of NFS, in many cases quite literally, considering the EFS protocol is a flavor of NFS. S3 buckets are just blobs with GUIDs in a flat namespace, which is literally what stateless NFS is--every "file" has a persistent UID (GUID if you assume host identifiers are unique), providing a simple handle for submitting idempotent block-oriented read and write operations. Theoretically, EFS could just be a fairly simple interface over S3, especially if you can implicitly wave away many of the caveats (e.g. wrt shared writes) by simply pointing out they have existed and mostly been tolerated in NFS environments for decades.
- geertj 4y ago
  (PM-T on the EFS team) S3 and EFS actually are quite different. Files on EFS are update-able, rename-able and link-able (I.e what’s expected from a file system), while S3 objects are immutable once they are created. This comes from the underlying data structures. EFS uses inodes and directories while S3 is more of a flat map. Protocol-wise EFS uses standard NFS 4.1. We added some optional innovations outside the protocol that you can use through our mount helper (mount.efs). This includes in-transit encryption with TLS (you can basically talk TLS to our endpoint and we will detect that automatically), and we support strong client auth using SigV4 over x509 client certificate.

mangix 4y ago

Isn't SMB better than NFS?

candiddevmike 4y ago
Depends on the use case. SMB auth is more robust and easier to integrate with AD, but NFS is simpler and typically faster for file access and transfer speeds. SMB is good for shares used by end users, NFS is good for shares used by services.
heavyset_go 4y ago
I've found NFSv4 to be more stable and performant than SMB when using it between Linux machines. Seems to handle multiple concurrent clients well, too.
- kevin_thibedeau 4y ago
  NFS works great until you lose the network and your client locks up.
  - heavyset_go 4y ago
    Mount your NFS shares with the `intr` and `vers=4.2` options.
  - teddyh 4y ago
    Are you really talking about NFSv4?
colordrops 4y ago
It at least doesn't lock anything up that has a file open when the network goes down. NFS is a nightmare with that. NFS is more idiomatic on *nix but still a huge pain when dealing with matching file perms across systems.
- dikei 4y ago
  > It at least doesn't lock anything up that has a file open when the network goes down. NFS is a nightmare with that. Yeah, we've been bitten by this too, around once a year, even with our fairly reliable and redundant network. It's a PITA, your process just hang and there's no way to even kill it except restarting the server.

pram 4y ago

My least favorite NFS thing I've dealt with was the integrated client in Oracle. Yes, really. Seemed almost impossible to figure out anything when stuff was broken. https://docs.oracle.com/en/database/oracle/oracle-database/19/ssdbi/about-direct-nfs-client-mounts-to-nfs-storage-devices.html#GUID-31591084-74BD-4B66-8C5B-68BF0FEE8750 https://docs.oracle.com/en/database/oracle/oracle-database/1...

brazzledazzle 4y ago
I had no idea this existed. This has to be considered masochism in software form.

netfortius 4y ago

Has AWS fixed the mounting of EFS under Windows based EC2s?

geertj 4y ago
We currently don't support Windows as a client. The main reason is that the builtin Windows NFS client is v3, while EFS is v4+. Some people have reported success mounting EFS on Windows using a third party NFS client, but I cannot comment on how well this works.

js2 4y ago

> One way is to generate a string that will be unique across all clients — possibly with host name, process ID, and timestamp — and then create a temporary file with this string as both name and content. This file is then (hard) linked to the name for the lock file. If the hard-link succeeds, the lock has been claimed. If it fails because the name already exists, then the application can read the content of that file. If it matches the generated unique string, then the error was due to a retransmit and again the lock has been claimed. Otherwise the application needs to sleep and try again. I seem to recall checking the link count on the temporary file is all that was needed: https://github.com/jaysoffian/dotlock https://github.com/jaysoffian/dotlock (That code is from 2001 or so.)

mprovost 4y ago

One of the reasons that NFS was so popular is that Sun gave away the protocol implementation. It's really interesting to see the technology behind that though. The whole protocol (both versions 2 and 3) is defined in a single file.[0] You run that file through the rpcgen program which spits out some .c and .h files which you can just link into your program. Of course on the server side you actually need to implement the code to handle those procedures, but on the client side it's all there, ready to use. [0] https://github.com/freebsd/freebsd-src/blob/main/include/rpcsvc/nfs_prot.x https://github.com/freebsd/freebsd-src/blob/main/include/rpc...

angrygoat 4y ago

A fun glitch with NFS with Linux serving out ext2/ext3 was getting -ENOENT when calling unlink() on a very large file (for the time.) The kernel on the server would do the work of unlinking the file which might take many seconds. In the meanwhile, the client would timeout, and make another NFS call to unlink. ext2/3 would have removed the path from the visible filesystem namespace, even though unlink hadn't complted, so this second call would return ENOENT. Somewhat confusing to users!

reacharavindh 4y ago

An unpopular opinion, but NFS is super handy and useful in reliable private networks with centralised authentication. Sure, it has its downsides that are being worked on with newer versions of the protocol(4+) with addition complexity, but it sure is useful in closely controlled setups like for HPC clusters. I ran a HPC cluster for an University, and relied upon good old NFSv3 for shared file storage(both home directories, and research datasets). In addition I also built out a big set of softwares compiled in one server and made available to the entire cluster via a read-only NFS mount point. The whole thing works so reliably without any hiccups whatsoever. To over some the limitations of authentication and authorisation with NFS storage, we use a centralised FreeIPA server that allows all machines in the cluster have the same UID/GID mapping everywhere. As a cream on top, the storage we expose over NFS is ZFS, that integrates nicely with NFS. Update 1: Yes, data security is a bit of an afterthought with NFS. As anybody in my network with physical access can mount my central storage to another server physically and access data as long as they can recreate UID/GID locally.. but, if I let someone to do this physically, I have bigger problems to deal with first.

teddyh 4y ago
You can fix the network access problem by using IPsec.
- throw0101a 4y ago
  NFS over TLS: * https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-rpc-tls https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-rpc-t...
  - teddyh 4y ago
    Latest draft expired more than a year ago?
    - throw0101a 4y ago
      Implementations for Linux and FreeBSD: * https://patchwork.kernel.org/project/cifs-client/cover/165030062272.5246.16956092606399079004.stgit@oracle-102.nfsv4.dev/ https://patchwork.kernel.org/project/cifs-client/cover/16503... * https://www.freshports.org/sysutils/nfs-over-tls/ https://www.freshports.org/sysutils/nfs-over-tls/ Activity on the NFSv4 mailing list: * https://mailarchive.ietf.org/arch/browse/nfsv4/ https://mailarchive.ietf.org/arch/browse/nfsv4/ But no recent commits to the draft: * https://github.com/chucklever/i-d-rpc-tls https://github.com/chucklever/i-d-rpc-tls ¯\_(ツ)_/¯

DonHopkins 4y ago

Another reason that NFS sucks: Anyone remember the Gator Box? It enabled you to trick NFS into putting slashes into the names of files and directories, which seemed to work at the time, but came back to totally fuck you later when you tried to restore a dump of your file system. The NFS protocol itself didn't disallow slashes in file names, so the NFS server would accept them without question from any client, silently corrupting the file system without any warning. Thanks, NFS! Oh and here's a great party trick that will totally blow your mind: On a Mac, use the Finder to create a folder or file whose name is today's date, like "2022/06/21", or anything with a slash in it. Cool, huh? Bet you didn't think you could do that! Now open a shell and "ls -l" the directory containing the file you just created with slashes in it name. What just happened there? Now try creating a folder or file whose name is the current time, or anything with colons, like "10:35:43". Ha ha! Don't worry, it's totally harmless and won't trash your file system or backups like NFS with a Gator Box would. https://news.ycombinator.com/item?id=20007875 https://news.ycombinator.com/item?id=20007875 DonHopkins on May 25, 2019 | parent | context | favorite | on: Why Does Windows Really Use Backslash as Path Sepa... There used to be a bug in the GatorBox Mac Localtalk-to-Ethernet NFS bridge that could somehow trick Unix into putting slashes into file names via NFS, which appeared to work fine, but then down the line Unix "restore" would totally shit itself. That was because Macs at the time (1991 or so) allowed you to use slashes (and spaces of course, but not colons, which it used a a path separator), and of course those silly Mac people, being touchy feely humans instead of hard core nerds, would dare to name files with dates like "My Spreadsheet 01/02/1991". https://en.wikipedia.org/wiki/GatorBox https://en.wikipedia.org/wiki/GatorBox Unix-Haters Handbook https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.txt https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.tx... Don't Touch That Slash! UFS allows any character in a filename except for the slash (/) and the ASCII NUL character. (Some versions of Unix allow ASCII characters with the high-bit, bit 8, set. Others don't.) This feature is great — especially in versions of Unix based on Berkeley's Fast File System, which allows filenames longer than 14 characters. It means that you are free to construct informative, easy-to-understand filenames like these: 1992 Sales Report Personnel File: Verne, Jules rt005mfkbgkw0 . cp Unfortunately, the rest of Unix isn't as tolerant. Of the filenames shown above, only rt005mfkbgkw0.cp will work with the majority of Unix utili- ties (which generally can't tolerate spaces in filenames). However, don't fret: Unix will let you construct filenames that have control characters or graphics symbols in them. (Some versions will even let you build files that have no name at all.) This can be a great security feature — especially if you have control keys on your keyboard that other people don't have on theirs. That's right: you can literally create files with names that other people can't access. It sort of makes up for the lack of serious security access controls in the rest of Unix. Recall that Unix does place one hard-and-fast restriction on filenames: they may never, ever contain the magic slash character (/), since the Unix kernel uses the slash to denote subdirectories. To enforce this requirement, the Unix kernel simply will never let you create a filename that has a slash in it. (However, you can have a filename with the 0200 bit set, which does list on some versions of Unix as a slash character.) Never? Well, hardly ever. Date: Mon, 8 Jan 90 18:41:57 PST From: sun!wrs!yuba!steve@decwrl.dec.com (Steve Sekiguchi) Subject: Info-Mac Digest V8 #3 5 I've got a rather difficult problem here. We've got a Gator Box run- ning the NFS/AFP conversion. We use this to hook up Macs and Suns. With the Sun as a AppleShare File server. All of this works great! Now here is the problem, Macs are allowed to create files on the Sun/ Unix fileserver with a "/" in the filename. This is great until you try to restore one of these files from your "dump" tapes, "restore" core dumps when it runs into a file with a "/" in the filename. As far as I can tell the "dump" tape is fine. Does anyone have a suggestion for getting the files off the backup tape? Thanks in Advance, Steven Sekiguchi Wind River Systems sun!wrs!steve, steve@wrs.com Emeryville CA, 94608 Apparently Sun's circa 1990 NFS server (which runs inside the kernel) assumed that an NFS client would never, ever send a filename that had a slash inside it and thus didn't bother to check for the illegal character. We're surprised that the files got written to the dump tape at all. (Then again, perhaps they didn't. There's really no way to tell for sure, is there now?)

JBiserkov 4y ago

Need for Speed

Someone 4y ago

I don’t understand the “as both the client and server must maintain the same list of locks for each client” part of: “NLM (the Network Lock Manager). This allows the client to request a byte-range lock on a given file (identified using an NFS file handle), and allows the server to grant it (or not), either immediately or later. Naturally this is an explicitly stateful protocol, as both the client and server must maintain the same list of locks for each client.”* There is no “the client”, so if clients have to maintain that information, how is it distributed to all clients (including those that will make their first request in the future)? How does the server even know all clients, given the statelessness of the protocol? Or does that locking only work for requests from the same server? Or does the client keep that information only so that it can unlock the ranges when it discovers the process that locked the range exits/crashed? Is it even correct to assume such range locks can’t be created by another process than the one that will delete them (say after the first process forked)?

infologics 4y ago

[dead]

rwmj 4y ago

The thing I hated in NFSv2/3 (and fixed, finally, in NFSv4) is the use of random port numbers, making it difficult to have an NFS server with a firewall. Yes you can set 5(!) environment variables in an obscure file to fix the port numbers. NFSv4 routes everything over port 2049, but unfortunately breaks user mapping, I guess we can't have everything.

jabl 4y ago
In the current NFSv4 RFC (https://datatracker.ietf.org/doc/html/rfc7530#section-5.9 https://datatracker.ietf.org/doc/html/rfc7530#section-5.9 ) it says: > To provide a greater degree of compatibility with NFSv3, which identified users and groups by 32-bit unsigned user identifiers and group identifiers, owner and group strings that consist of ASCII- encoded decimal numeric values with no leading zeros can be given a special interpretation by clients and servers that choose to provide such support. The receiver may treat such a user or group string as representing the same user as would be represented by an NFSv3 uid or gid having the corresponding numeric value. I'm not sure how common this extension is, at least the Linux server and client support it out of the box. Isilon also supports it, but it must be explicitly enabled.
- geertj 4y ago
  > I'm not sure how common this extension is, at least the Linux server and client support it out of the box. Isilon also supports it, but it must be explicitly enabled. It is very common. I’m not aware of a v4 server that does not support this. Originally v4 went all in on Kerberos (GSSAPI technically) to provide strong multi user auth. This is the reason that users and groups are represented as strings. This approach works reasonably well on Windows with SMB since you have AD there giving you Kerberos and a shared (LDAP) directory. AD is also deeply integrated in the OS for things like credential management and provisioning. The approach did not work so well on Linux where not everyone is running AD or even some kind of directory. This caused the protocol designers to make Kerberos optional in v4.1. I guess the spec authors already knew that Kerberos was going to be difficult because I just checked and the numerical-strong-user-as-posix-id workaround was already present in the original 4.0 spec.

irusensei 4y ago

Interesting fact: Linux kernel.org website used to offer public over the internet NFS (and SMB) mirrors. https://web.archive.org/web/19980130085039/http://www.kernel.org/ https://web.archive.org/web/19980130085039/http://www.kernel...

chasil 4y ago
Some of the big public FTP servers also did this. I think I remember it on one of the sunsite servers.

atulsnj 4y ago

Silly me, I was hoping for a game review.

atulsnj 4y ago

Silly me, I clicked hoping for a game review.

bluedino 4y ago

NFS can be...interesting to configure. Most of your enterprise storage systems support it (as well as the smaller stuff like Synology) and there's all kinds of 'but what about if we need to...' https://www.delltechnologies.com/asset/en-us/products/storage/industry-market/h17240_wp_isilon_onefs_nfs_design_considerations_bp.pdf https://www.delltechnologies.com/asset/en-us/products/storag...

smcameron 4y ago

I noticed that NFS became a lot less important to me around the time that distributed version control software (specifically CVS) became popular. Suddenly the main reason I was using NFS, to share our RCS source code repo, disappeared.

lizknope 4y ago

Did anybody use Auspex NFS servers in the late 1990's? My first job in 1997 was all Sun / SPARC based (semiconductor design) and we had a bunch of Auspex file servers. It looks like they went under in 2003 and NetApp became the dominant company.

DonHopkins 4y ago
Guy Harris worked at Sun from 1985 - 1988, then on NFS at Auspex from 1988 - 1994, then he worked at NetApp from 1994 - 2003. After that he worked at Apple, now he's been hacking Wireshark for more than 23 years, since 1998! https://ask.wireshark.org/users/12/guy-harris/ https://ask.wireshark.org/users/12/guy-harris/ https://www.wireshark.org/docs/wsug_html_chunked/ChIntroHistory.html https://www.wireshark.org/docs/wsug_html_chunked/ChIntroHist...
acdha 4y ago
We got a lot of value out of ours in an academic research context but switched around 2002 when they just couldn't manage to be price-competitive. We ended up with Spinnaker, Isilon, and NetApp boxes which were all Linux-based, although with some custom extensions (if memory serves, we found a bunch of correctness issues running fsx against Isilon because their filesystem implementation wasn't as robust as the stock Linux filesystems).