9 ms·

NFS at 40 – Remembering the Sun Microsystems Network File System

178 points by signa11 1y ago

holoduke 1y ago
We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challenge though.
- rootnod3 1y ago
  True. But for example a home server I absolutely love the simplicity. I have 6 Lenovo 720q machines, one of them as a data storage just running simple NFS for quick daily backups before it pushes them to a NAS.
- hnlmorg 1y ago
  Unfortunately there doesn’t seem to be any decent alternative. SMB is a nightmare to set up if your host isn’t running Windows. sshfs is actually pretty good but it’s not exactly ubiquitous. Plus it has its own quirks and performs slower. So it really doesn’t feel like an upgrade. Everything else I know of is either proprietary, or hard to set up. Or both. These days everything has gone more cloud-oriented. Eg Dropbox et al. And I don’t want to sync with a cloud server just to sync between two local machines.
  - jjtheblunt 1y ago
    What happened to Transarc's DFS ? I looked, found the link below, but it seems to just fizzle out without info. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System Anyway, we used it extensively in the UIUC engineering workstation labs hundreds of computers, 20+ years ago, and it worked excellently. I set up a server farm 20 years ago of Sun sparcs but used NFS for such.
    - convolvatron 1y ago
      I used to administer AFS/DFS and braved the forest of platform ifdefs to port it to different unix flavors. plusses were security (kerberos), better administrative controls and global file space. minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful. I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.
      - jjtheblunt 1y ago
        that's very similar to what we were doing for the engineering workstations (hundreds of hosts across a very fast campus network) (off topic, but great username)
    - nbernard 1y ago
      AFS (on which DFS was based) lives on as OpenAFS [0]. And there is a commercial evolution/solution from AuriStor [1]. [0]: https://openafs.org/ https://openafs.org/ [1]: https://www.auristor.com/filesystem/ https://www.auristor.com/filesystem/
      - jjtheblunt 1y ago
        Thank you! I knew it was from Andrew File System but did not manage to find those links by searching too narrowly.
        jaltman 1y ago
        DCE DFS (developed at Transarc) was originally supposed to be AFS 4.0 before it was contributed to DCE. After the contribution it became backward incompatible with AFS 3.x. The RPC layer, the authentication protocol, the protection service (user/group management) were all replaced to leverage technology contributions from other DCE participants. IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure. There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System
        jjtheblunt 1y ago
        thanks : i hadn't known that larger context either, so very useful.
  - toast0 1y ago
    > SMB is a nightmare to set up if your host isn’t running Windows. Samba runs fine on my FreeBSD host? All my clients are Windows though. If I wanted to have a non-windows desktop client, I'd probably use NFS for the same share.
    - hnlmorg 1y ago
      It runs fine but it's a nightmare to set up. It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly. It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon. Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.
  - fodkodrasz 1y ago
    SMB is not that terrible to set up (has its quirks definitely), but apple devices don't interoperate well in my experience. SMB from my samba server performs very well from linux and windows clients alike, but the performance from mac is terrible. NFS support was lacking on windows when I last tried. I used NFS (v3) a lot in the past, but unless in a highly static high trust environment, it was worse to use than SMB (for me). Especially the user-id mapping story is something I'm not sure is solved properly. That was a PITA in the homelab scale, having to set up NIS was really something I didn't like, a road warrior setup didn't work well for me, I quickly abandoned it.
    - hnlmorg 1y ago
      > SMB is not that terrible to set up Samba can be. Especially when compared with NFS > NFS support was lacking on windows when I last tried. If you need to connect from Windows then your options are very limited, unfortunately.
    - Spooky23 1y ago
      Windows 10/11 have native support. Writes aren’t terribly performant iirc.
    - chasil 1y ago
      Which SMB? SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions. SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only. SMBv3 supports optional encryption. Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option. NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.
      - p_ing 1y ago
        For macOS, it doesn't matter which protocol version of SMB, it's just a poor implementation of SMB [client].
  - NexRebular 1y ago
    > SMB is a nightmare to set up if your host isn’t running Windows. It's very easy on illumos based systems due the integrated SMB/CIFS service.
  - Spivak 1y ago
    I mean the decent alternative is object storage if you can tolerate not getting a filesystem. You can get an S3 client running anywhere with little trouble. There are lots of really good S3 compatible servers you can self-host. And you don't get the issue of your system locking up because of an unresponsive server. I've always thought that NFS makes you choose between two bad alternatives with "stop the world and wait" or "fail in a way that apps are not prepared for."
    - hnlmorg 1y ago
      If you don't need a filesystem, then your options are numerous. The problem is sometimes you do need exactly that. I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.
      - zokier 1y ago
        AWS has this "mountpoint for s3" thingy https://github.com/awslabs/mountpoint-s3 https://github.com/awslabs/mountpoint-s3
        hnlmorg 1y ago
        They do, but POSIX file system APIs don’t map to S3 APIs well. So you run the risk of heavily increasing your S3 APIs costs for any Stat() heavy workflows. This is why I say there’s mixed opinions about mounting S3 via FUSE. This isn’t an issue with a self hosted S3 compatible storage server. But you then have potential issues using an AWS tool for non-AWS infra. There be dragons there. And if you where to use a 3rd party S3 mounting tool, then you run into all the other read and write performance issues that they had (and why Amazon ended up writing their own tool for S3). So it’s really not a trivial exercise to selfhost a mountable block storage server. And for something as important as data consistency, you might well be concerned enough about weird edge cases that mature technologies like SMB and NFS just feel safer.
  - lmm 1y ago
    > SMB is a nightmare to set up if your host isn’t running Windows. That's the opposite of my experience. Fire it up and it just works, in less time than it would take you to configure NFS sensibly.
    - hnlmorg 1y ago
      Have you ever had to manually administer a samba daemon?
      - lmm 1y ago
        Yes, and I've also had to manually administer an NFS daemon. I know which I prefer.
        hnlmorg 1y ago
        Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Samba can be set up easily enough if you know what you’re doing. But getting the AD controller part working would often throw up annoying edge case problems. Problems that I never had to deal with in NFS. Though I will admit that NIS/YP could be a pain if you needed it to sync with NT.
        apelapan 1y ago
        NIS/YP on its own were amazing though! So simple to setup and did exactly everything you could ever need in a small to medium-sized network. Everything that was supposed to replace it is so much worse, except for supposedly not being very unsafe.
        lmm 1y ago
        > Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Might just be bad timing then, most of my experience with it was in that v3/v4 transition period. It was bad enough to make me swear off the whole thing.
  - ForHackernews 1y ago
    Someone else in this thread was suggesting http://www.openafs.org/ http://www.openafs.org/ but I had never heard of it.
- q3k 1y ago
  9P? Significantly simpler, at the protocol level, than NFS (to the point where you can implement a client/server in your language of choice in one afternoon).
- jabl 1y ago
  Lustre is big in the HPC/AI training world. Amazing performance and scalability, but not for the faint of heart.
  - nwellinghoff 1y ago
    Got any more details about pros and cons based on your experience?
    - jabl 1y ago
      Well, kind of hard to say anything exhaustive in a quick comment, but roughly advantages: - POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent. - performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days. - Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems. Downsides: - Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then. - When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different. To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_report_marco_passerini_lad_2024.pdf https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster-love-and-hate.pdf https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...
mixmastamyk 1y ago
What are most people using today for file serving? For our little lan sftp seems adequate, since ssh is already running.
- pkulak 1y ago
  SMB has always worked great for me.
- Arubis 1y ago
  NFS! At least on my localnet.
- nine_k 1y ago
  SMB2 for high-performance writable shares, WebDAV for high-performance read-only shares, also firewall-friendly. Sftp is useful, but is pretty slow, only good for small amounts and small number of files. (Or maybe i don't know how to cook it properly.)
  - aborsy 1y ago
    SMB is great for LAN, but its performance over internet is poor. It remains SFTP and WebDAV in that case. SFTP would be my choice, if there is client support.
    - nine_k 1y ago
      I suspect that NFS over Internet is also not the most brilliant idea; I assumed the LAN setting.
- Narushia 1y ago
  NFS v4.2. Easy to set up if you don't need authentication. Very good throughput, at least so long as your network gear isn't the bottleneck. I think it's the best choice if your clients are Linux or similar. The only bummer for me is that mounting NFS shares from Android file managers seems to be difficult or impossible (let alone NFSv4).
ryandrake 1y ago
NFS is the backbone of my home network servers, including file sharing (books, movies, music), local backups, source code and development, and large volumes of data for hobby projects. I don't know what I'd do without it. Haven't found anything more suitable in 15+ years.
- INTPenis 1y ago
  Same. The latest thing I did was put snes state and save files on NFS so I can resume the same game from laptop, to retropi (tv), and even on the road over wireguard.
01HNNWZ0MV43FF 1y ago
I'd seen a proposal to use loopback NFS in place of FUSE: https://github.com/xetdata/nfsserve https://github.com/xetdata/nfsserve
- toomim 1y ago
  See also https://www.legitcontrol.com https://www.legitcontrol.com as presented at https://braid.org/meeting-118 https://braid.org/meeting-118 for a beautiful example of "local NFS" as a wonderful replacement for FUSE!
AshamedCaptain 1y ago
> There is also a site, nfsv4bat.org [...] However, be careful: the site is insecure I just find this highly ironic considering this is NFS we are talking about. Also, do they fear their ISPs changing the 40 year old NFS specs on the flight or what ? Why even mention this ?
buserror 1y ago
I use NFS as a keystone of a pretty large multi-million data center application. I run it on a dedicated 100Gb network with 9k frames and it works fantastic. I'm pretty sure it is still use in many, many places because... it works! I don't need to "remember NFS", NFS is a big part of my day!
- zh3 1y ago
  On a smaller scale, I run multiple PC's in house diskless with NFS root; so easy to just create copies on the server and boot into them as needed, it's almost one image per bloated app these days (server also boots PC's into Windows using iSCSI/SCST and old DOS boxes from 386 onwards with etherboot/samba). Probably a bit biased due to doing a lot of hardware hacking where virtualisation solutions take so much more effect, but got to agree NFS (from V2 through V4) just works.
semi-extrinsic 1y ago
I'm considering NFS with RDMA for a handful of CFD workstations + one file server with 25Gbe network. Anyone know if this will perform well? Will be using XFS with some NVME disks as the base FS on the file server.
- fock 1y ago
  Quite some time ago I implemented NFS for a small HPC-cluster on a 40GBe network. A colleague set up RDMA later, since at start it didn't work with the Ubuntu kernel available. Full nVME on the file server too. While the raw performance using ZFS was kind of underwhelming (mdadm+XFS about 2x faster), network performance was fine I'd argue: serial transfers easily hit ~4GB/s on a single node and 4K-benchmarking with fio was comparable to a good SATA-SSD (IOPS + throughput) on multiple clients in parallel!
- heavyset_go 1y ago
  Yes, you might want to tune your NFS parameters, stick to NFSv4.2, consider if caching is appropriate for your workloads and at what level, and how much of your NFS + networking you can keep in kernel space if you decide to further upgrade your network's throughput or really expand it. Also consider what your server and client machines will be running, some NFS clients suck. Linux on both ends works really well.
  - chasil 1y ago
    The last time that I looked, OpenBSD does not support NFSv4 at all.
- SoftTalker 1y ago
  Consider BeeGFS. Had good results with it using infiniband.
jedberg 1y ago
My NFS story: In my first job, we used NFS to maintain the developer desktops. They were all FreeBSD and remote mounted /usr/local. This worked great! Everyone worked in the office with fast local internet, and it made it easy for us to add or update apps and have everyone magically get it. And when the NFS server had a glitch, our devs could usually just reboot and fix it, or wait a bit. Since they were all systems developers they all understood the problems with NFS and the workarounds. What I learned though was that NFS was great until it wasn't. If the server hung, all work stopped. When I got to reddit, solving code distribution was one of the first tasks I had to take care of. Steve wanted to use NFS to distribute the app code. He wanted to have all the app servers mount an NFS mount, and then just update the code there and have them all automatically pick up the changes. This sounded great in theory, but I told him about all the gotchas. He didn't believe me, so I pulled up a bunch of papers and blog posts, and actually set up a small cluster to show him what happens when the server goes offline, and how none of the app servers could keep running as soon as they had to get anything off disk. To his great credit, he trusted me after that when I said something was a bad idea based on my experience. It was an important lesson for me that even with experience, trust must be earned when you work with a new team. I set up a system where app servers would pull fresh code on boot and we could also remotely trigger a pull or just push to them, and that system was reddit's deployment tool for about a decade (and it was written in Perl!)
- zh3 1y ago
  Don't know about FreeBSD but hard hanging on a mounted filesystem is configurable (if it's essential configure it that way, otherwise don't). To this day I see plenty of code written that hangs forever if a remote resource is unavailable.
  - muxator 1y ago
    Hi, could you give some pointers about this? Thanks!
    - zh3 1y ago
      It's down to the mount options, use 'soft' and the program trying to access the (inaccessible) server gets an error return after a while, or 'intr' if you want to be able to kill the hung process. The caveat is a lot of software is written to assume things like fread(), fopen() etc will either quickly fail or work. However, if the file is over a network obviously things can go wrong so the common default behaviour is to wait for the server to come back online. Same issue applies to any other network filesystem, different OS's (and even the same OS with different configs) handle the situation differently.
cramcgrab 1y ago
Auto home! And jumpstart! Aah, the network is the computer!
cramcgrab 1y ago
Zfs includes nfs, its built in and very handy still!
- E39M5S62 1y ago
  If you're talking about OpenZFS, that is a thin wrapper over knfsd/exports file. They don't actually ship an NFS daemon in the OpenZFS code.
- krunger 1y ago
  ZFS is amazing, used it since around Solaris 10, and yes, loved it for it's NFS capability, had many terabytes at the time on it, back when a terabyte meant a rack of drives! Now those same systems host petabytes, all upgraded in place. Solaris was pretty amazing too.
sunshine-o 1y ago
If only I could mount a NFS share Android ...
- Narushia 1y ago
  I looked into this a while ago and was surprised to find that no file explorer on Android seems to support it[1]. However, I did notice that VLC for Android does support it, though unfortunately only NFSv3. I was at least able to watch some videos from the share with it, but it would be nice to have general access to the share on Android. [1] Of course, I didn’t test every single app — there’s a bucketload of them on Google Play and elsewhere…
  - sunshine-o 1y ago
    Yes ! they are using a a client library for that https://github.com/sahlberg/libnfs https://github.com/sahlberg/libnfs
    - Narushia 1y ago
      Interesting, the readme for that library says that NFSv4 is supported. So that likely means that VLC is doing something wrong on their side, because only NFSv3 works?
- heavyset_go 1y ago
  Been a while, but if you root your phone and have access to the kernel source in order to build the NFS modules, would you be able to mount NFS shares then?
  - sunshine-o 1y ago
    Yes, you need a rooted phone since usually support is at the kernel level
    - heavyset_go 1y ago
      Nice, glad it's still an option
Eikon 1y ago
ZeroFS uses NFS/9P instead of fuse! https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS
nasretdinov 1y ago
I have really mixed feelings about things like NFS, remote desktop, etc. The idea of having everything remote to save resources (or for other reasons) does sound really appealing in theory, and, when it works, is truly great. However in practice it's really hard to make these things be worth it, because of latency. E.g. for network block storage and for NFS the performance is usually abysmal compared to even a relatively cheap modern SSD in terms of latency, and many applications now expect a low latency file system, and perform really poorly otherwise.
- zh3 1y ago
  Fairly obviously a 1Gbps network is not going to compete with 5Gbps SATA or 20Gbps NVME. Having said that, for real performance we load stuff over the network into local RAM and then generally run from that (faster than all other options). On the internal network the server also has a large RAM disk shared over NFS/SMB, and the performance PC's have plenty of RAM - so really it's a tradeoff, and the optimum is going to depend on how the system is used.
  - Palomides 1y ago
    want to emphasize, for those who haven't been following, a nice used 25Gb ethernet card is like $25 now
    - zokier 1y ago
      But how much is a 25GbE (or 40GbE) switch?
      - Palomides 1y ago
        maybe around $400 for 25Gbe depending on your noise and power tolerance, and 40Gbe is dirt cheap now if you only have two or three devices that need a fast connection you can just do point to point, of course
      - bombcar 1y ago
        10gb is cheap as free → CRS304-4XG-IN If I needed more than that, I’d probably do a direct link.
E39M5S62 1y ago
PornHub's origin clusters serve petabytes of files off of NFS mounts - it's still alive and well in lots of places.
commandersaki 1y ago
Ah Network Failure System, good memories.
amacbride 1y ago
My introduction to NFS was first at Berkeley, and then at Sun. It more or less just worked. (Some of the early file servers at Berkeley were drastically overcapacity with all the diskless Sun-3/50s connected, but still.) And of course I still use it every day with Amazon EFS; Happy Birthday, indeed!
cyberax 1y ago
A good time to plug my NFSv4 client in Go: https://github.com/Cyberax/go-nfs-client https://github.com/Cyberax/go-nfs-client :) It's made for EFS, but works well enough with other servers.
alexpotato 1y ago
CTO at a hedge fund I worked at had a great quote: "NFS is like heroin: it seems like a great idea at first and then it ruins your life" (as many commenters are pointing out) Still an amazing technology for it's time though.
brcmthrowaway 1y ago
How does Isilon compare to nfs?
- green-salt 1y ago
  Isilon has their own filesystem that stores the data across multi-node clusters, you then export that out over NFS/SMB/S3, and the nodes load balance the I/O across the cluster.
technofiend 1y ago
As a unix sysadmin in the early 90s, I liked to understand as much as I could about the tech that supported the systems I supported. All my clients used NFS so I dug into the guts of RPC until I could write my own services and publish them via portmap. Weirdly that nerd snipe landed me two different jobs! People wanted to build network-based services and that was one of the quickest ways to do it.
la64710 1y ago
Autofs Still magic …
pjmlp 1y ago
The wonders of not being able to login on university servers due to NFS communication problems preventing to mount the directory. Usually caused by a coaxial cable not being properly terminated. Naturally it meant nothing on the network was working, however NFS was kind of the canary in the mine for it.
axpvms 1y ago
I liked this post about No File Security from a previous thread https://news.ycombinator.com/item?id=31820504 https://news.ycombinator.com/item?id=31820504
cik 1y ago
I still love NFS. It's a cornerstone to how I end up thinking about many problems. In my house it provides a simple NAS mount. In certain development environments, I use sshmount because of it. But I really loved the lesser known RFS. Yes it wasn't as robust, or as elegent.. but there's nothing quite like mounting someone else's sound card and blaring music out of it, in order to drive a prank. Sigh...
- ozozozd 1y ago
  Like someone’s _Sound Blaster_?
lukeh 1y ago
It was amusing to read the comment about the flat UID/GID namespace being a problem, identified as far back as 1984. This is something that DCE addressed by using a larger namespace (UUIDs), and Windows finally got right using a hierarchical one (SIDs).
jamespo 1y ago
Anyone remember https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS ?
irusensei 1y ago
I think NFS is the most sane storage provider for self hosted Kubernetes. Anything else seems over engineered for a home lab and is normally not a very good experience. What I don't like is the security model. It's either deploying kerberos infrastructure or "trust me bro I'm UID 1000" so I default to SMB on the file server.
drewg123 1y ago
NFS was a huge part of my life in the 90s.. - It caused me to switch from Linux to FreeBSD in 1994 when Linux didn't have NFS caching but FreeBSD did & Linus told me "nobody cares about NFS" at the Boston USENIX. I was a sysadmin for a small stats department, and they made heavy use of NFS mounted latex fonts. Xdvi would render a page in 1s on FreeBSD and over a minute on Linux due to the difference in caching. Xdvi would seek byte-by-byte in the file.. You could see each character as it rendered on Linux, and the page would just open instantly on FreeBSD. - When working as research staff for the OS research group in the CS dept, I worked on a modern cluster filesystem ("slice") which pretended to be NFS for client compat. (https://www.usenix.org/legacy/event/osdi00/full_papers/andersonchase/andersonchase.pdf https://www.usenix.org/legacy/event/osdi00/full_papers/ander...)
Publius_Enigma 1y ago
Still performs better than SMB on macOS for many workloads!
- p_ing 1y ago
  Apple just doesn't seem to care about SMB on macOS even though it is now the primary file sharing protocol.
msravi 1y ago
Wait. I still export NFS mounts from my TrueNAS server and make them available to all other machines on my LAN (music, books, documents, photos, etc). The article and comments here give me the feeling that NFS is outadated and shouldn't be used anymore. Am I doing things wrong?
- SamPatt 1y ago
  I do exactly the same thing, and it works beautifully. We can't really be doing it wrong if it's working!
Publius_Enigma 1y ago
I’ve been using NFS in various environments since my first introduction to it in my university’s Solaris and Linux labs. I’ve run it at home, on and off, since 2005. I’ve recently started using it again after consistent issues with SMB on Apple devices, and the deprecation of AFP. My FreeBSD server, running on a Raspberry Pi, makes terabytes of content available to the web via an NFS connection to a Synology NAS. For my use case, with a small number of users, the fact that NFS is host based rather than user based, means I can set it up one on each device, and all users of that host can access the shares. And I’ve generally found it to be more consistently performant on Apple hardware than their in-house SMB implementation.
mannyv 1y ago
NFS, hard mounts, hang. I remember them well.

178 points by signa11 1y ago

We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challenge though.

rootnod3 1y ago
True. But for example a home server I absolutely love the simplicity. I have 6 Lenovo 720q machines, one of them as a data storage just running simple NFS for quick daily backups before it pushes them to a NAS.
hnlmorg 1y ago
Unfortunately there doesn’t seem to be any decent alternative. SMB is a nightmare to set up if your host isn’t running Windows. sshfs is actually pretty good but it’s not exactly ubiquitous. Plus it has its own quirks and performs slower. So it really doesn’t feel like an upgrade. Everything else I know of is either proprietary, or hard to set up. Or both. These days everything has gone more cloud-oriented. Eg Dropbox et al. And I don’t want to sync with a cloud server just to sync between two local machines.
- jjtheblunt 1y ago
  What happened to Transarc's DFS ? I looked, found the link below, but it seems to just fizzle out without info. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System Anyway, we used it extensively in the UIUC engineering workstation labs hundreds of computers, 20+ years ago, and it worked excellently. I set up a server farm 20 years ago of Sun sparcs but used NFS for such.
  - convolvatron 1y ago
    I used to administer AFS/DFS and braved the forest of platform ifdefs to port it to different unix flavors. plusses were security (kerberos), better administrative controls and global file space. minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful. I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.
    - jjtheblunt 1y ago
      that's very similar to what we were doing for the engineering workstations (hundreds of hosts across a very fast campus network) (off topic, but great username)
  - nbernard 1y ago
    AFS (on which DFS was based) lives on as OpenAFS [0]. And there is a commercial evolution/solution from AuriStor [1]. [0]: https://openafs.org/ https://openafs.org/ [1]: https://www.auristor.com/filesystem/ https://www.auristor.com/filesystem/
    - jjtheblunt 1y ago
      Thank you! I knew it was from Andrew File System but did not manage to find those links by searching too narrowly.
      - jaltman 1y ago
        DCE DFS (developed at Transarc) was originally supposed to be AFS 4.0 before it was contributed to DCE. After the contribution it became backward incompatible with AFS 3.x. The RPC layer, the authentication protocol, the protection service (user/group management) were all replaced to leverage technology contributions from other DCE participants. IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure. There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System
        jjtheblunt 1y ago
        thanks : i hadn't known that larger context either, so very useful.
- toast0 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. Samba runs fine on my FreeBSD host? All my clients are Windows though. If I wanted to have a non-windows desktop client, I'd probably use NFS for the same share.
  - hnlmorg 1y ago
    It runs fine but it's a nightmare to set up. It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly. It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon. Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.
- fodkodrasz 1y ago
  SMB is not that terrible to set up (has its quirks definitely), but apple devices don't interoperate well in my experience. SMB from my samba server performs very well from linux and windows clients alike, but the performance from mac is terrible. NFS support was lacking on windows when I last tried. I used NFS (v3) a lot in the past, but unless in a highly static high trust environment, it was worse to use than SMB (for me). Especially the user-id mapping story is something I'm not sure is solved properly. That was a PITA in the homelab scale, having to set up NIS was really something I didn't like, a road warrior setup didn't work well for me, I quickly abandoned it.
  - hnlmorg 1y ago
    > SMB is not that terrible to set up Samba can be. Especially when compared with NFS > NFS support was lacking on windows when I last tried. If you need to connect from Windows then your options are very limited, unfortunately.
  - Spooky23 1y ago
    Windows 10/11 have native support. Writes aren’t terribly performant iirc.
  - chasil 1y ago
    Which SMB? SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions. SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only. SMBv3 supports optional encryption. Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option. NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.
    - p_ing 1y ago
      For macOS, it doesn't matter which protocol version of SMB, it's just a poor implementation of SMB [client].
- NexRebular 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. It's very easy on illumos based systems due the integrated SMB/CIFS service.
- Spivak 1y ago
  I mean the decent alternative is object storage if you can tolerate not getting a filesystem. You can get an S3 client running anywhere with little trouble. There are lots of really good S3 compatible servers you can self-host. And you don't get the issue of your system locking up because of an unresponsive server. I've always thought that NFS makes you choose between two bad alternatives with "stop the world and wait" or "fail in a way that apps are not prepared for."
  - hnlmorg 1y ago
    If you don't need a filesystem, then your options are numerous. The problem is sometimes you do need exactly that. I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.
    - zokier 1y ago
      AWS has this "mountpoint for s3" thingy https://github.com/awslabs/mountpoint-s3 https://github.com/awslabs/mountpoint-s3
      - hnlmorg 1y ago
        They do, but POSIX file system APIs don’t map to S3 APIs well. So you run the risk of heavily increasing your S3 APIs costs for any Stat() heavy workflows. This is why I say there’s mixed opinions about mounting S3 via FUSE. This isn’t an issue with a self hosted S3 compatible storage server. But you then have potential issues using an AWS tool for non-AWS infra. There be dragons there. And if you where to use a 3rd party S3 mounting tool, then you run into all the other read and write performance issues that they had (and why Amazon ended up writing their own tool for S3). So it’s really not a trivial exercise to selfhost a mountable block storage server. And for something as important as data consistency, you might well be concerned enough about weird edge cases that mature technologies like SMB and NFS just feel safer.
- lmm 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. That's the opposite of my experience. Fire it up and it just works, in less time than it would take you to configure NFS sensibly.
  - hnlmorg 1y ago
    Have you ever had to manually administer a samba daemon?
    - lmm 1y ago
      Yes, and I've also had to manually administer an NFS daemon. I know which I prefer.
      - hnlmorg 1y ago
        Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Samba can be set up easily enough if you know what you’re doing. But getting the AD controller part working would often throw up annoying edge case problems. Problems that I never had to deal with in NFS. Though I will admit that NIS/YP could be a pain if you needed it to sync with NT.
        apelapan 1y ago
        NIS/YP on its own were amazing though! So simple to setup and did exactly everything you could ever need in a small to medium-sized network. Everything that was supposed to replace it is so much worse, except for supposedly not being very unsafe.
        lmm 1y ago
        > Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Might just be bad timing then, most of my experience with it was in that v3/v4 transition period. It was bad enough to make me swear off the whole thing.
- ForHackernews 1y ago
  Someone else in this thread was suggesting http://www.openafs.org/ http://www.openafs.org/ but I had never heard of it.
q3k 1y ago
9P? Significantly simpler, at the protocol level, than NFS (to the point where you can implement a client/server in your language of choice in one afternoon).
jabl 1y ago
Lustre is big in the HPC/AI training world. Amazing performance and scalability, but not for the faint of heart.
- nwellinghoff 1y ago
  Got any more details about pros and cons based on your experience?
  - jabl 1y ago
    Well, kind of hard to say anything exhaustive in a quick comment, but roughly advantages: - POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent. - performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days. - Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems. Downsides: - Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then. - When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different. To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_report_marco_passerini_lad_2024.pdf https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster-love-and-hate.pdf https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...

mixmastamyk 1y ago

What are most people using today for file serving? For our little lan sftp seems adequate, since ssh is already running.

pkulak 1y ago
SMB has always worked great for me.
Arubis 1y ago
NFS! At least on my localnet.
nine_k 1y ago
SMB2 for high-performance writable shares, WebDAV for high-performance read-only shares, also firewall-friendly. Sftp is useful, but is pretty slow, only good for small amounts and small number of files. (Or maybe i don't know how to cook it properly.)
- aborsy 1y ago
  SMB is great for LAN, but its performance over internet is poor. It remains SFTP and WebDAV in that case. SFTP would be my choice, if there is client support.
  - nine_k 1y ago
    I suspect that NFS over Internet is also not the most brilliant idea; I assumed the LAN setting.
Narushia 1y ago
NFS v4.2. Easy to set up if you don't need authentication. Very good throughput, at least so long as your network gear isn't the bottleneck. I think it's the best choice if your clients are Linux or similar. The only bummer for me is that mounting NFS shares from Android file managers seems to be difficult or impossible (let alone NFSv4).

ryandrake 1y ago

NFS is the backbone of my home network servers, including file sharing (books, movies, music), local backups, source code and development, and large volumes of data for hobby projects. I don't know what I'd do without it. Haven't found anything more suitable in 15+ years.

INTPenis 1y ago
Same. The latest thing I did was put snes state and save files on NFS so I can resume the same game from laptop, to retropi (tv), and even on the road over wireguard.

01HNNWZ0MV43FF 1y ago

I'd seen a proposal to use loopback NFS in place of FUSE: https://github.com/xetdata/nfsserve https://github.com/xetdata/nfsserve

toomim 1y ago
See also https://www.legitcontrol.com https://www.legitcontrol.com as presented at https://braid.org/meeting-118 https://braid.org/meeting-118 for a beautiful example of "local NFS" as a wonderful replacement for FUSE!

AshamedCaptain 1y ago

> There is also a site, nfsv4bat.org [...] However, be careful: the site is insecure I just find this highly ironic considering this is NFS we are talking about. Also, do they fear their ISPs changing the 40 year old NFS specs on the flight or what ? Why even mention this ?

buserror 1y ago

I use NFS as a keystone of a pretty large multi-million data center application. I run it on a dedicated 100Gb network with 9k frames and it works fantastic. I'm pretty sure it is still use in many, many places because... it works! I don't need to "remember NFS", NFS is a big part of my day!

zh3 1y ago
On a smaller scale, I run multiple PC's in house diskless with NFS root; so easy to just create copies on the server and boot into them as needed, it's almost one image per bloated app these days (server also boots PC's into Windows using iSCSI/SCST and old DOS boxes from 386 onwards with etherboot/samba). Probably a bit biased due to doing a lot of hardware hacking where virtualisation solutions take so much more effect, but got to agree NFS (from V2 through V4) just works.

semi-extrinsic 1y ago

I'm considering NFS with RDMA for a handful of CFD workstations + one file server with 25Gbe network. Anyone know if this will perform well? Will be using XFS with some NVME disks as the base FS on the file server.

fock 1y ago
Quite some time ago I implemented NFS for a small HPC-cluster on a 40GBe network. A colleague set up RDMA later, since at start it didn't work with the Ubuntu kernel available. Full nVME on the file server too. While the raw performance using ZFS was kind of underwhelming (mdadm+XFS about 2x faster), network performance was fine I'd argue: serial transfers easily hit ~4GB/s on a single node and 4K-benchmarking with fio was comparable to a good SATA-SSD (IOPS + throughput) on multiple clients in parallel!
heavyset_go 1y ago
Yes, you might want to tune your NFS parameters, stick to NFSv4.2, consider if caching is appropriate for your workloads and at what level, and how much of your NFS + networking you can keep in kernel space if you decide to further upgrade your network's throughput or really expand it. Also consider what your server and client machines will be running, some NFS clients suck. Linux on both ends works really well.
- chasil 1y ago
  The last time that I looked, OpenBSD does not support NFSv4 at all.
SoftTalker 1y ago
Consider BeeGFS. Had good results with it using infiniband.

jedberg 1y ago

My NFS story: In my first job, we used NFS to maintain the developer desktops. They were all FreeBSD and remote mounted /usr/local. This worked great! Everyone worked in the office with fast local internet, and it made it easy for us to add or update apps and have everyone magically get it. And when the NFS server had a glitch, our devs could usually just reboot and fix it, or wait a bit. Since they were all systems developers they all understood the problems with NFS and the workarounds. What I learned though was that NFS was great until it wasn't. If the server hung, all work stopped. When I got to reddit, solving code distribution was one of the first tasks I had to take care of. Steve wanted to use NFS to distribute the app code. He wanted to have all the app servers mount an NFS mount, and then just update the code there and have them all automatically pick up the changes. This sounded great in theory, but I told him about all the gotchas. He didn't believe me, so I pulled up a bunch of papers and blog posts, and actually set up a small cluster to show him what happens when the server goes offline, and how none of the app servers could keep running as soon as they had to get anything off disk. To his great credit, he trusted me after that when I said something was a bad idea based on my experience. It was an important lesson for me that even with experience, trust must be earned when you work with a new team. I set up a system where app servers would pull fresh code on boot and we could also remotely trigger a pull or just push to them, and that system was reddit's deployment tool for about a decade (and it was written in Perl!)

zh3 1y ago
Don't know about FreeBSD but hard hanging on a mounted filesystem is configurable (if it's essential configure it that way, otherwise don't). To this day I see plenty of code written that hangs forever if a remote resource is unavailable.
- muxator 1y ago
  Hi, could you give some pointers about this? Thanks!
  - zh3 1y ago
    It's down to the mount options, use 'soft' and the program trying to access the (inaccessible) server gets an error return after a while, or 'intr' if you want to be able to kill the hung process. The caveat is a lot of software is written to assume things like fread(), fopen() etc will either quickly fail or work. However, if the file is over a network obviously things can go wrong so the common default behaviour is to wait for the server to come back online. Same issue applies to any other network filesystem, different OS's (and even the same OS with different configs) handle the situation differently.

cramcgrab 1y ago

Auto home! And jumpstart! Aah, the network is the computer!

cramcgrab 1y ago

Zfs includes nfs, its built in and very handy still!

E39M5S62 1y ago
If you're talking about OpenZFS, that is a thin wrapper over knfsd/exports file. They don't actually ship an NFS daemon in the OpenZFS code.
krunger 1y ago
ZFS is amazing, used it since around Solaris 10, and yes, loved it for it's NFS capability, had many terabytes at the time on it, back when a terabyte meant a rack of drives! Now those same systems host petabytes, all upgraded in place. Solaris was pretty amazing too.

sunshine-o 1y ago

If only I could mount a NFS share Android ...

Narushia 1y ago
I looked into this a while ago and was surprised to find that no file explorer on Android seems to support it[1]. However, I did notice that VLC for Android does support it, though unfortunately only NFSv3. I was at least able to watch some videos from the share with it, but it would be nice to have general access to the share on Android. [1] Of course, I didn’t test every single app — there’s a bucketload of them on Google Play and elsewhere…
- sunshine-o 1y ago
  Yes ! they are using a a client library for that https://github.com/sahlberg/libnfs https://github.com/sahlberg/libnfs
  - Narushia 1y ago
    Interesting, the readme for that library says that NFSv4 is supported. So that likely means that VLC is doing something wrong on their side, because only NFSv3 works?
heavyset_go 1y ago
Been a while, but if you root your phone and have access to the kernel source in order to build the NFS modules, would you be able to mount NFS shares then?
- sunshine-o 1y ago
  Yes, you need a rooted phone since usually support is at the kernel level
  - heavyset_go 1y ago
    Nice, glad it's still an option

Eikon 1y ago

ZeroFS uses NFS/9P instead of fuse! https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS

nasretdinov 1y ago

I have really mixed feelings about things like NFS, remote desktop, etc. The idea of having everything remote to save resources (or for other reasons) does sound really appealing in theory, and, when it works, is truly great. However in practice it's really hard to make these things be worth it, because of latency. E.g. for network block storage and for NFS the performance is usually abysmal compared to even a relatively cheap modern SSD in terms of latency, and many applications now expect a low latency file system, and perform really poorly otherwise.

zh3 1y ago
Fairly obviously a 1Gbps network is not going to compete with 5Gbps SATA or 20Gbps NVME. Having said that, for real performance we load stuff over the network into local RAM and then generally run from that (faster than all other options). On the internal network the server also has a large RAM disk shared over NFS/SMB, and the performance PC's have plenty of RAM - so really it's a tradeoff, and the optimum is going to depend on how the system is used.
- Palomides 1y ago
  want to emphasize, for those who haven't been following, a nice used 25Gb ethernet card is like $25 now
  - zokier 1y ago
    But how much is a 25GbE (or 40GbE) switch?
    - Palomides 1y ago
      maybe around $400 for 25Gbe depending on your noise and power tolerance, and 40Gbe is dirt cheap now if you only have two or three devices that need a fast connection you can just do point to point, of course
    - bombcar 1y ago
      10gb is cheap as free → CRS304-4XG-IN If I needed more than that, I’d probably do a direct link.

E39M5S62 1y ago

PornHub's origin clusters serve petabytes of files off of NFS mounts - it's still alive and well in lots of places.

commandersaki 1y ago

Ah Network Failure System, good memories.

amacbride 1y ago

My introduction to NFS was first at Berkeley, and then at Sun. It more or less just worked. (Some of the early file servers at Berkeley were drastically overcapacity with all the diskless Sun-3/50s connected, but still.) And of course I still use it every day with Amazon EFS; Happy Birthday, indeed!

cyberax 1y ago

A good time to plug my NFSv4 client in Go: https://github.com/Cyberax/go-nfs-client https://github.com/Cyberax/go-nfs-client :) It's made for EFS, but works well enough with other servers.

alexpotato 1y ago

CTO at a hedge fund I worked at had a great quote: "NFS is like heroin: it seems like a great idea at first and then it ruins your life" (as many commenters are pointing out) Still an amazing technology for it's time though.

brcmthrowaway 1y ago

How does Isilon compare to nfs?

green-salt 1y ago
Isilon has their own filesystem that stores the data across multi-node clusters, you then export that out over NFS/SMB/S3, and the nodes load balance the I/O across the cluster.

technofiend 1y ago

As a unix sysadmin in the early 90s, I liked to understand as much as I could about the tech that supported the systems I supported. All my clients used NFS so I dug into the guts of RPC until I could write my own services and publish them via portmap. Weirdly that nerd snipe landed me two different jobs! People wanted to build network-based services and that was one of the quickest ways to do it.

la64710 1y ago

Autofs Still magic …

pjmlp 1y ago

The wonders of not being able to login on university servers due to NFS communication problems preventing to mount the directory. Usually caused by a coaxial cable not being properly terminated. Naturally it meant nothing on the network was working, however NFS was kind of the canary in the mine for it.

axpvms 1y ago

I liked this post about No File Security from a previous thread https://news.ycombinator.com/item?id=31820504 https://news.ycombinator.com/item?id=31820504

cik 1y ago

I still love NFS. It's a cornerstone to how I end up thinking about many problems. In my house it provides a simple NAS mount. In certain development environments, I use sshmount because of it. But I really loved the lesser known RFS. Yes it wasn't as robust, or as elegent.. but there's nothing quite like mounting someone else's sound card and blaring music out of it, in order to drive a prank. Sigh...

ozozozd 1y ago
Like someone’s _Sound Blaster_?

lukeh 1y ago

It was amusing to read the comment about the flat UID/GID namespace being a problem, identified as far back as 1984. This is something that DCE addressed by using a larger namespace (UUIDs), and Windows finally got right using a hierarchical one (SIDs).

jamespo 1y ago

Anyone remember https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS ?

irusensei 1y ago

I think NFS is the most sane storage provider for self hosted Kubernetes. Anything else seems over engineered for a home lab and is normally not a very good experience. What I don't like is the security model. It's either deploying kerberos infrastructure or "trust me bro I'm UID 1000" so I default to SMB on the file server.

drewg123 1y ago

NFS was a huge part of my life in the 90s.. - It caused me to switch from Linux to FreeBSD in 1994 when Linux didn't have NFS caching but FreeBSD did & Linus told me "nobody cares about NFS" at the Boston USENIX. I was a sysadmin for a small stats department, and they made heavy use of NFS mounted latex fonts. Xdvi would render a page in 1s on FreeBSD and over a minute on Linux due to the difference in caching. Xdvi would seek byte-by-byte in the file.. You could see each character as it rendered on Linux, and the page would just open instantly on FreeBSD. - When working as research staff for the OS research group in the CS dept, I worked on a modern cluster filesystem ("slice") which pretended to be NFS for client compat. (https://www.usenix.org/legacy/event/osdi00/full_papers/andersonchase/andersonchase.pdf https://www.usenix.org/legacy/event/osdi00/full_papers/ander...)

Publius_Enigma 1y ago

Still performs better than SMB on macOS for many workloads!

p_ing 1y ago
Apple just doesn't seem to care about SMB on macOS even though it is now the primary file sharing protocol.

msravi 1y ago

Wait. I still export NFS mounts from my TrueNAS server and make them available to all other machines on my LAN (music, books, documents, photos, etc). The article and comments here give me the feeling that NFS is outadated and shouldn't be used anymore. Am I doing things wrong?

SamPatt 1y ago
I do exactly the same thing, and it works beautifully. We can't really be doing it wrong if it's working!

Publius_Enigma 1y ago

I’ve been using NFS in various environments since my first introduction to it in my university’s Solaris and Linux labs. I’ve run it at home, on and off, since 2005. I’ve recently started using it again after consistent issues with SMB on Apple devices, and the deprecation of AFP. My FreeBSD server, running on a Raspberry Pi, makes terabytes of content available to the web via an NFS connection to a Synology NAS. For my use case, with a small number of users, the fact that NFS is host based rather than user based, means I can set it up one on each device, and all users of that host can access the shares. And I’ve generally found it to be more consistently performant on Apple hardware than their in-house SMB implementation.

mannyv 1y ago

NFS, hard mounts, hang. I remember them well.