5 ms·

Start with the second answer: don't use network filesystems for production, period. To explain that you explain the first answer: network filesystems are flaky,

Start with the second answer: don't use network filesystems for production, period. To explain that you explain the first answer: network filesystems are flaky, applications and operations that depend on them often come up with weird bugs, they aren't very fast, they aren't very secure, they introduce problematic dependencies and problematic design patterns. There are locking issues, time sync issues, permission mapping issues, there are bugs in kernel and usermode drivers, there are conflicts between clients and servers from different vendors, there are bugs in implementations of standards (and different standard versions), there are varying degrees of support, there is always an intermittent network outage that causes hard to detect bugs, you have to decide between hard and soft process locking for i/o threads, performance turning for either throughput or speed, sync or async, export security rules, encryption or no encryption (and does everything you use support it), to say nothing of dedicated or shared network gear, finding an admin to run your expensive enterprise gear implementing the server (open source nfs implementations are a joke and lack critical features), or basic maintenance like expanding storage or replication. The list goes on and on.

mjg59 7y ago
And yet Facebook appear to be successfully running a somewhat popular website while using NFS in production, which implies that the situation is somewhat less absolute than you claim.
- peterwwillis 7y ago
  Oh, you can definitely make money with a design from 2005 that uses NFS in production, 4 layers of caching, and enough servers that rebooting them constantly doesn't impact your service. I've worked at those companies. It was a bad idea then, it's a bad idea now.
  - richardwhiuk 7y ago
    Is there anything that's a good idea? Or is this just "everything has bugs and issues"?
    - deleted 7y ago
      [deleted]
    - notacoward 7y ago
      Seems like the latter. I'd guess peterwwillis has had some bad experiences with NFS. I've had some bad experience with some of the same so-called alternatives. Every time I see someone put a text blob into a database that needs to be in a file at point of use, I want to slap them. Ditto for everyone who tries to implement a hierarchical namespace on top of a flat blob store. Ditto for everyone who distributes a thousand copies of a rarely-changing file via scp or chef, then complains about update speed or inconsistency. NFS is not generally a good idea in high-volume high-node-count traffic paths, but it can be a less-bad solution in many other situations including many that are still part of production.
  - notacoward 7y ago
    That's not how it's done, and I know because I was in that group. Also, note that "production" at a company like Facebook includes a lot more than the direct web-content-serving path. I think I might agree that NFS doesn't make sense in that particular path, because it provides features and guarantees that nothing in that path needs and that means needless complexity, but there are plenty of other roles where NFS is a better fit than blob stores or copying files around.
    - peterwwillis 7y ago
      And there's plenty of other solutions that don't have all the problems (and overhead) NFS has. There are precious few actual good reasons to use NFS and they aren't customer facing products. Particularly, when you want a rootless thin terminal on an embedded system, or read-only transfer of non-sensitive files (say, for bootstrapping a machine from bootp, or sharing a corpus of data). For everything else (in production), NFS is a PITA, and most other networked filesystems should be avoided for similar reasons. Again: other solutions exist that provide the same functionality but do it without the inherent problems. Usually they are ignored because someone wanted to cut cost, or because legacy. Ask a storage engineer or infrastructure architect.
      - notacoward 7y ago
        > Ask a storage engineer or infrastructure architect. I've been both, and known hundreds. I'd love to hear about this solution you have that has the same functionality (your words) as a network/distributed filesystem, same compatibility, same semantic richness, same consistency, etc. without any of the tradeoffs. What are you selling?
        peterwwillis 7y ago
        Well to start, virtually any networked filesystem is better. Even CIFS won't hose your system from network blips, but there's also GFS2, HDFS, GPFS, OCFS, VMFS, Ceph, Lustre, Gluter, Lizard, Orange, Hammer2, MapR, Xtreem. If you don't want to roll your own OSS solution (and I wouldn't recommend it) most modern SANs have all those features and more, available via a variety of interconnects and protocols (including shared volumes). Real replication, real snapshots, real encryption, real RBACs, and not going over the same interface as random network traffic. Storage manufacturers even offer custom drivers for K8s or VMware so you can control the persistent storage for your cloud apps directly from your orchestrator's control plane, or manage it in your SAN.

mjg59 7y ago

And yet Facebook appear to be successfully running a somewhat popular website while using NFS in production, which implies that the situation is somewhat less absolute than you claim.

peterwwillis 7y ago
Oh, you can definitely make money with a design from 2005 that uses NFS in production, 4 layers of caching, and enough servers that rebooting them constantly doesn't impact your service. I've worked at those companies. It was a bad idea then, it's a bad idea now.
- richardwhiuk 7y ago
  Is there anything that's a good idea? Or is this just "everything has bugs and issues"?
  - deleted 7y ago
    [deleted]
  - notacoward 7y ago
    Seems like the latter. I'd guess peterwwillis has had some bad experiences with NFS. I've had some bad experience with some of the same so-called alternatives. Every time I see someone put a text blob into a database that needs to be in a file at point of use, I want to slap them. Ditto for everyone who tries to implement a hierarchical namespace on top of a flat blob store. Ditto for everyone who distributes a thousand copies of a rarely-changing file via scp or chef, then complains about update speed or inconsistency. NFS is not generally a good idea in high-volume high-node-count traffic paths, but it can be a less-bad solution in many other situations including many that are still part of production.
- notacoward 7y ago
  That's not how it's done, and I know because I was in that group. Also, note that "production" at a company like Facebook includes a lot more than the direct web-content-serving path. I think I might agree that NFS doesn't make sense in that particular path, because it provides features and guarantees that nothing in that path needs and that means needless complexity, but there are plenty of other roles where NFS is a better fit than blob stores or copying files around.
  - peterwwillis 7y ago
    And there's plenty of other solutions that don't have all the problems (and overhead) NFS has. There are precious few actual good reasons to use NFS and they aren't customer facing products. Particularly, when you want a rootless thin terminal on an embedded system, or read-only transfer of non-sensitive files (say, for bootstrapping a machine from bootp, or sharing a corpus of data). For everything else (in production), NFS is a PITA, and most other networked filesystems should be avoided for similar reasons. Again: other solutions exist that provide the same functionality but do it without the inherent problems. Usually they are ignored because someone wanted to cut cost, or because legacy. Ask a storage engineer or infrastructure architect.
    - notacoward 7y ago
      > Ask a storage engineer or infrastructure architect. I've been both, and known hundreds. I'd love to hear about this solution you have that has the same functionality (your words) as a network/distributed filesystem, same compatibility, same semantic richness, same consistency, etc. without any of the tradeoffs. What are you selling?
      - peterwwillis 7y ago
        Well to start, virtually any networked filesystem is better. Even CIFS won't hose your system from network blips, but there's also GFS2, HDFS, GPFS, OCFS, VMFS, Ceph, Lustre, Gluter, Lizard, Orange, Hammer2, MapR, Xtreem. If you don't want to roll your own OSS solution (and I wouldn't recommend it) most modern SANs have all those features and more, available via a variety of interconnects and protocols (including shared volumes). Real replication, real snapshots, real encryption, real RBACs, and not going over the same interface as random network traffic. Storage manufacturers even offer custom drivers for K8s or VMware so you can control the persistent storage for your cloud apps directly from your orchestrator's control plane, or manage it in your SAN.