5 ms·

We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challe

We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challenge though.

rootnod3 1y ago
True. But for example a home server I absolutely love the simplicity. I have 6 Lenovo 720q machines, one of them as a data storage just running simple NFS for quick daily backups before it pushes them to a NAS.
hnlmorg 1y ago
Unfortunately there doesn’t seem to be any decent alternative. SMB is a nightmare to set up if your host isn’t running Windows. sshfs is actually pretty good but it’s not exactly ubiquitous. Plus it has its own quirks and performs slower. So it really doesn’t feel like an upgrade. Everything else I know of is either proprietary, or hard to set up. Or both. These days everything has gone more cloud-oriented. Eg Dropbox et al. And I don’t want to sync with a cloud server just to sync between two local machines.
- jjtheblunt 1y ago
  What happened to Transarc's DFS ? I looked, found the link below, but it seems to just fizzle out without info. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System Anyway, we used it extensively in the UIUC engineering workstation labs hundreds of computers, 20+ years ago, and it worked excellently. I set up a server farm 20 years ago of Sun sparcs but used NFS for such.
  - convolvatron 1y ago
    I used to administer AFS/DFS and braved the forest of platform ifdefs to port it to different unix flavors. plusses were security (kerberos), better administrative controls and global file space. minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful. I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.
    - jjtheblunt 1y ago
      that's very similar to what we were doing for the engineering workstations (hundreds of hosts across a very fast campus network) (off topic, but great username)
  - nbernard 1y ago
    AFS (on which DFS was based) lives on as OpenAFS [0]. And there is a commercial evolution/solution from AuriStor [1]. [0]: https://openafs.org/ https://openafs.org/ [1]: https://www.auristor.com/filesystem/ https://www.auristor.com/filesystem/
    - jjtheblunt 1y ago
      Thank you! I knew it was from Andrew File System but did not manage to find those links by searching too narrowly.
      - jaltman 1y ago
        DCE DFS (developed at Transarc) was originally supposed to be AFS 4.0 before it was contributed to DCE. After the contribution it became backward incompatible with AFS 3.x. The RPC layer, the authentication protocol, the protection service (user/group management) were all replaced to leverage technology contributions from other DCE participants. IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure. There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System
        jjtheblunt 1y ago
        thanks : i hadn't known that larger context either, so very useful.
- toast0 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. Samba runs fine on my FreeBSD host? All my clients are Windows though. If I wanted to have a non-windows desktop client, I'd probably use NFS for the same share.
  - hnlmorg 1y ago
    It runs fine but it's a nightmare to set up. It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly. It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon. Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.
- fodkodrasz 1y ago
  SMB is not that terrible to set up (has its quirks definitely), but apple devices don't interoperate well in my experience. SMB from my samba server performs very well from linux and windows clients alike, but the performance from mac is terrible. NFS support was lacking on windows when I last tried. I used NFS (v3) a lot in the past, but unless in a highly static high trust environment, it was worse to use than SMB (for me). Especially the user-id mapping story is something I'm not sure is solved properly. That was a PITA in the homelab scale, having to set up NIS was really something I didn't like, a road warrior setup didn't work well for me, I quickly abandoned it.
  - hnlmorg 1y ago
    > SMB is not that terrible to set up Samba can be. Especially when compared with NFS > NFS support was lacking on windows when I last tried. If you need to connect from Windows then your options are very limited, unfortunately.
  - Spooky23 1y ago
    Windows 10/11 have native support. Writes aren’t terribly performant iirc.
  - chasil 1y ago
    Which SMB? SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions. SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only. SMBv3 supports optional encryption. Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option. NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.
    - p_ing 1y ago
      For macOS, it doesn't matter which protocol version of SMB, it's just a poor implementation of SMB [client].
- NexRebular 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. It's very easy on illumos based systems due the integrated SMB/CIFS service.
- Spivak 1y ago
  I mean the decent alternative is object storage if you can tolerate not getting a filesystem. You can get an S3 client running anywhere with little trouble. There are lots of really good S3 compatible servers you can self-host. And you don't get the issue of your system locking up because of an unresponsive server. I've always thought that NFS makes you choose between two bad alternatives with "stop the world and wait" or "fail in a way that apps are not prepared for."
  - hnlmorg 1y ago
    If you don't need a filesystem, then your options are numerous. The problem is sometimes you do need exactly that. I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.
    - zokier 1y ago
      AWS has this "mountpoint for s3" thingy https://github.com/awslabs/mountpoint-s3 https://github.com/awslabs/mountpoint-s3
      - hnlmorg 1y ago
        They do, but POSIX file system APIs don’t map to S3 APIs well. So you run the risk of heavily increasing your S3 APIs costs for any Stat() heavy workflows. This is why I say there’s mixed opinions about mounting S3 via FUSE. This isn’t an issue with a self hosted S3 compatible storage server. But you then have potential issues using an AWS tool for non-AWS infra. There be dragons there. And if you where to use a 3rd party S3 mounting tool, then you run into all the other read and write performance issues that they had (and why Amazon ended up writing their own tool for S3). So it’s really not a trivial exercise to selfhost a mountable block storage server. And for something as important as data consistency, you might well be concerned enough about weird edge cases that mature technologies like SMB and NFS just feel safer.
- lmm 1y ago
  > SMB is a nightmare to set up if your host isn’t running Windows. That's the opposite of my experience. Fire it up and it just works, in less time than it would take you to configure NFS sensibly.
  - hnlmorg 1y ago
    Have you ever had to manually administer a samba daemon?
    - lmm 1y ago
      Yes, and I've also had to manually administer an NFS daemon. I know which I prefer.
      - hnlmorg 1y ago
        Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Samba can be set up easily enough if you know what you’re doing. But getting the AD controller part working would often throw up annoying edge case problems. Problems that I never had to deal with in NFS. Though I will admit that NIS/YP could be a pain if you needed it to sync with NT.
        apelapan 1y ago
        NIS/YP on its own were amazing though! So simple to setup and did exactly everything you could ever need in a small to medium-sized network. Everything that was supposed to replace it is so much worse, except for supposedly not being very unsafe.
        lmm 1y ago
        > Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Might just be bad timing then, most of my experience with it was in that v3/v4 transition period. It was bad enough to make me swear off the whole thing.
- ForHackernews 1y ago
  Someone else in this thread was suggesting http://www.openafs.org/ http://www.openafs.org/ but I had never heard of it.
q3k 1y ago
9P? Significantly simpler, at the protocol level, than NFS (to the point where you can implement a client/server in your language of choice in one afternoon).
jabl 1y ago
Lustre is big in the HPC/AI training world. Amazing performance and scalability, but not for the faint of heart.
- nwellinghoff 1y ago
  Got any more details about pros and cons based on your experience?
  - jabl 1y ago
    Well, kind of hard to say anything exhaustive in a quick comment, but roughly advantages: - POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent. - performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days. - Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems. Downsides: - Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then. - When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different. To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_report_marco_passerini_lad_2024.pdf https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster-love-and-hate.pdf https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...

We are still using it for some pretty large apps. Still have not found a good and simple alternative. I like the simplicity and performance. Scaling is a challenge though.

rootnod3 1y ago

True. But for example a home server I absolutely love the simplicity. I have 6 Lenovo 720q machines, one of them as a data storage just running simple NFS for quick daily backups before it pushes them to a NAS.

hnlmorg 1y ago

Unfortunately there doesn’t seem to be any decent alternative. SMB is a nightmare to set up if your host isn’t running Windows. sshfs is actually pretty good but it’s not exactly ubiquitous. Plus it has its own quirks and performs slower. So it really doesn’t feel like an upgrade. Everything else I know of is either proprietary, or hard to set up. Or both. These days everything has gone more cloud-oriented. Eg Dropbox et al. And I don’t want to sync with a cloud server just to sync between two local machines.

jjtheblunt 1y ago
What happened to Transarc's DFS ? I looked, found the link below, but it seems to just fizzle out without info. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System Anyway, we used it extensively in the UIUC engineering workstation labs hundreds of computers, 20+ years ago, and it worked excellently. I set up a server farm 20 years ago of Sun sparcs but used NFS for such.
- convolvatron 1y ago
  I used to administer AFS/DFS and braved the forest of platform ifdefs to port it to different unix flavors. plusses were security (kerberos), better administrative controls and global file space. minuses were generally poor performance, middling small file support and awful large file support. substantial administrative overhead. the wide-area performance was so bad the global namespace thing wasn't really useful. I guess it didn't cause as many actual multi-hour outages NFS, but we used it primarily for home/working directories and left the servers alone, whereas the accepted practice at the time was to use NFS for roots and to cross mount everything so that it easily got into a 'help I've fallen and can't get up' situation.
  - jjtheblunt 1y ago
    that's very similar to what we were doing for the engineering workstations (hundreds of hosts across a very fast campus network) (off topic, but great username)
- nbernard 1y ago
  AFS (on which DFS was based) lives on as OpenAFS [0]. And there is a commercial evolution/solution from AuriStor [1]. [0]: https://openafs.org/ https://openafs.org/ [1]: https://www.auristor.com/filesystem/ https://www.auristor.com/filesystem/
  - jjtheblunt 1y ago
    Thank you! I knew it was from Andrew File System but did not manage to find those links by searching too narrowly.
    - jaltman 1y ago
      DCE DFS (developed at Transarc) was originally supposed to be AFS 4.0 before it was contributed to DCE. After the contribution it became backward incompatible with AFS 3.x. The RPC layer, the authentication protocol, the protection service (user/group management) were all replaced to leverage technology contributions from other DCE participants. IMO IBM/Transarc died for two reasons. First, there was significant brand confusion after the release of Windows Active Directory and Windows DFS since no trademarks were obtained for DCE service names. Second, the file system couldn't be deployed without the rest of the DCE infrastructure. There was an unofficial effort within IBM to create the Advanced Distributed File System (ADFS) which would have decoupled DFS from the DCE Cell Directory Service and Security Service as well as replaced DCE/RPC. However, the project never saw the light of day. https://en.wikipedia.org/wiki/DCE_Distributed_File_System https://en.wikipedia.org/wiki/DCE_Distributed_File_System
      - jjtheblunt 1y ago
        thanks : i hadn't known that larger context either, so very useful.
toast0 1y ago
> SMB is a nightmare to set up if your host isn’t running Windows. Samba runs fine on my FreeBSD host? All my clients are Windows though. If I wanted to have a non-windows desktop client, I'd probably use NFS for the same share.
- hnlmorg 1y ago
  It runs fine but it's a nightmare to set up. It's one of those tools that, unless you already know what you're doing, you can expect to sink several hours into trying to get the damn thing working correctly. It's not the kind of thing you can throw at a junior and expect them to get working in an afternoon. Whereas NFS and sshfs "just work". Albeit I will concede that NFSv4 was annoying to get working back when that was new too. But that's, thankfully, a distant memory.
fodkodrasz 1y ago
SMB is not that terrible to set up (has its quirks definitely), but apple devices don't interoperate well in my experience. SMB from my samba server performs very well from linux and windows clients alike, but the performance from mac is terrible. NFS support was lacking on windows when I last tried. I used NFS (v3) a lot in the past, but unless in a highly static high trust environment, it was worse to use than SMB (for me). Especially the user-id mapping story is something I'm not sure is solved properly. That was a PITA in the homelab scale, having to set up NIS was really something I didn't like, a road warrior setup didn't work well for me, I quickly abandoned it.
- hnlmorg 1y ago
  > SMB is not that terrible to set up Samba can be. Especially when compared with NFS > NFS support was lacking on windows when I last tried. If you need to connect from Windows then your options are very limited, unfortunately.
- Spooky23 1y ago
  Windows 10/11 have native support. Writes aren’t terribly performant iirc.
- chasil 1y ago
  Which SMB? SMBv1 has a reputation for being an extremely chatty protocol. Novell ipx/spx easily dwarfed it in the early days. Microsoft now disables it by default, but some utilities (curl) do not support more advanced versions. SMBv2 increases efficiency by bundling multiple messages into a single transmission. It is clear text only. SMBv3 supports optional encryption. Apple dropped the Samba project from MacOS due to gplv3, and developed their own SMB implementation that is not used elsewhere AFAIK. If you don't care for Apple's implementation, then perhaps installing Samba is a better option. NFSv3 relies solely on uid/gid mapping by default, while NFSv4 requires idmapd to run to avoid squashing. I sometimes use both at the same time.
  - p_ing 1y ago
    For macOS, it doesn't matter which protocol version of SMB, it's just a poor implementation of SMB [client].
NexRebular 1y ago
> SMB is a nightmare to set up if your host isn’t running Windows. It's very easy on illumos based systems due the integrated SMB/CIFS service.
Spivak 1y ago
I mean the decent alternative is object storage if you can tolerate not getting a filesystem. You can get an S3 client running anywhere with little trouble. There are lots of really good S3 compatible servers you can self-host. And you don't get the issue of your system locking up because of an unresponsive server. I've always thought that NFS makes you choose between two bad alternatives with "stop the world and wait" or "fail in a way that apps are not prepared for."
- hnlmorg 1y ago
  If you don't need a filesystem, then your options are numerous. The problem is sometimes you do need exactly that. I do agree that object storage is a nice option. I wonder if a FUSE-like object storage wrapper would work well here. I've seen mixed results for S3 but for local instances, it might be a different story.
  - zokier 1y ago
    AWS has this "mountpoint for s3" thingy https://github.com/awslabs/mountpoint-s3 https://github.com/awslabs/mountpoint-s3
    - hnlmorg 1y ago
      They do, but POSIX file system APIs don’t map to S3 APIs well. So you run the risk of heavily increasing your S3 APIs costs for any Stat() heavy workflows. This is why I say there’s mixed opinions about mounting S3 via FUSE. This isn’t an issue with a self hosted S3 compatible storage server. But you then have potential issues using an AWS tool for non-AWS infra. There be dragons there. And if you where to use a 3rd party S3 mounting tool, then you run into all the other read and write performance issues that they had (and why Amazon ended up writing their own tool for S3). So it’s really not a trivial exercise to selfhost a mountable block storage server. And for something as important as data consistency, you might well be concerned enough about weird edge cases that mature technologies like SMB and NFS just feel safer.
lmm 1y ago
> SMB is a nightmare to set up if your host isn’t running Windows. That's the opposite of my experience. Fire it up and it just works, in less time than it would take you to configure NFS sensibly.
- hnlmorg 1y ago
  Have you ever had to manually administer a samba daemon?
  - lmm 1y ago
    Yes, and I've also had to manually administer an NFS daemon. I know which I prefer.
    - hnlmorg 1y ago
      Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Samba can be set up easily enough if you know what you’re doing. But getting the AD controller part working would often throw up annoying edge case problems. Problems that I never had to deal with in NFS. Though I will admit that NIS/YP could be a pain if you needed it to sync with NT.
      - apelapan 1y ago
        NIS/YP on its own were amazing though! So simple to setup and did exactly everything you could ever need in a small to medium-sized network. Everything that was supposed to replace it is so much worse, except for supposedly not being very unsafe.
      - lmm 1y ago
        > Weird. I’ve done both at scale many times and NFS daemons have always been significantly less problematic (bar the brief period when NFSv4 was new, but I just fell back to NFSv3 for a brief period). Might just be bad timing then, most of my experience with it was in that v3/v4 transition period. It was bad enough to make me swear off the whole thing.
ForHackernews 1y ago
Someone else in this thread was suggesting http://www.openafs.org/ http://www.openafs.org/ but I had never heard of it.

q3k 1y ago

9P? Significantly simpler, at the protocol level, than NFS (to the point where you can implement a client/server in your language of choice in one afternoon).

jabl 1y ago

Lustre is big in the HPC/AI training world. Amazing performance and scalability, but not for the faint of heart.

nwellinghoff 1y ago
Got any more details about pros and cons based on your experience?
- jabl 1y ago
  Well, kind of hard to say anything exhaustive in a quick comment, but roughly advantages: - POSIX compliant, including dotting the i's. As opposed to, say, NFS which isn't cache coherent. - performance and scalability. 1 TB/s+ sequential IO to a single file is what you'd expect on a large HPC system these days. - Metadata performance has gotten a lot better over the past decade or so, beating most(all?) other parallel filesystems. Downsides: - Lots of pieces in a Lustre cluster (typically nodes are paired in sort-of active/active HA configs). And lots of cables, switches etc. So a fairly decent chance something breaks every now and then. - When something breaks, Lustre is weird and different compared to many other filesystems. Tools are rudimentary and different. To get a feel for what 'life with Lustre' could be, see e.g. various 'site reports' from workshops. E.g. for a couple somewhat recent ones: https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_report_marco_passerini_lad_2024.pdf https://www.eofs.eu/wp-content/uploads/2024/09/cscs_site_rep... and https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster-love-and-hate.pdf https://www.eofs.eu/wp-content/uploads/2024/09/LAD-24-Luster...