5 ms·

Interesting comment, but for those of use who don't do infra but are curious, what is the problem with NFS, and what do you use instead? My ears always prick up

by mc3 7y ago

Interesting comment, but for those of use who don't do infra but are curious, what is the problem with NFS, and what do you use instead? My ears always prick up when someone says ... or implies ... to not use something in production!

peterwwillis 7y ago
Start with the second answer: don't use network filesystems for production, period. To explain that you explain the first answer: network filesystems are flaky, applications and operations that depend on them often come up with weird bugs, they aren't very fast, they aren't very secure, they introduce problematic dependencies and problematic design patterns. There are locking issues, time sync issues, permission mapping issues, there are bugs in kernel and usermode drivers, there are conflicts between clients and servers from different vendors, there are bugs in implementations of standards (and different standard versions), there are varying degrees of support, there is always an intermittent network outage that causes hard to detect bugs, you have to decide between hard and soft process locking for i/o threads, performance turning for either throughput or speed, sync or async, export security rules, encryption or no encryption (and does everything you use support it), to say nothing of dedicated or shared network gear, finding an admin to run your expensive enterprise gear implementing the server (open source nfs implementations are a joke and lack critical features), or basic maintenance like expanding storage or replication. The list goes on and on.
- mjg59 7y ago
  And yet Facebook appear to be successfully running a somewhat popular website while using NFS in production, which implies that the situation is somewhat less absolute than you claim.
  - peterwwillis 7y ago
    Oh, you can definitely make money with a design from 2005 that uses NFS in production, 4 layers of caching, and enough servers that rebooting them constantly doesn't impact your service. I've worked at those companies. It was a bad idea then, it's a bad idea now.
    - richardwhiuk 7y ago
      Is there anything that's a good idea? Or is this just "everything has bugs and issues"?
      - deleted 7y ago
        [deleted]
      - notacoward 7y ago
        Seems like the latter. I'd guess peterwwillis has had some bad experiences with NFS. I've had some bad experience with some of the same so-called alternatives. Every time I see someone put a text blob into a database that needs to be in a file at point of use, I want to slap them. Ditto for everyone who tries to implement a hierarchical namespace on top of a flat blob store. Ditto for everyone who distributes a thousand copies of a rarely-changing file via scp or chef, then complains about update speed or inconsistency. NFS is not generally a good idea in high-volume high-node-count traffic paths, but it can be a less-bad solution in many other situations including many that are still part of production.
    - notacoward 7y ago
      That's not how it's done, and I know because I was in that group. Also, note that "production" at a company like Facebook includes a lot more than the direct web-content-serving path. I think I might agree that NFS doesn't make sense in that particular path, because it provides features and guarantees that nothing in that path needs and that means needless complexity, but there are plenty of other roles where NFS is a better fit than blob stores or copying files around.
      - peterwwillis 7y ago
        And there's plenty of other solutions that don't have all the problems (and overhead) NFS has. There are precious few actual good reasons to use NFS and they aren't customer facing products. Particularly, when you want a rootless thin terminal on an embedded system, or read-only transfer of non-sensitive files (say, for bootstrapping a machine from bootp, or sharing a corpus of data). For everything else (in production), NFS is a PITA, and most other networked filesystems should be avoided for similar reasons. Again: other solutions exist that provide the same functionality but do it without the inherent problems. Usually they are ignored because someone wanted to cut cost, or because legacy. Ask a storage engineer or infrastructure architect.

by mc3 7y ago

peterwwillis 7y ago

Start with the second answer: don't use network filesystems for production, period. To explain that you explain the first answer: network filesystems are flaky, applications and operations that depend on them often come up with weird bugs, they aren't very fast, they aren't very secure, they introduce problematic dependencies and problematic design patterns. There are locking issues, time sync issues, permission mapping issues, there are bugs in kernel and usermode drivers, there are conflicts between clients and servers from different vendors, there are bugs in implementations of standards (and different standard versions), there are varying degrees of support, there is always an intermittent network outage that causes hard to detect bugs, you have to decide between hard and soft process locking for i/o threads, performance turning for either throughput or speed, sync or async, export security rules, encryption or no encryption (and does everything you use support it), to say nothing of dedicated or shared network gear, finding an admin to run your expensive enterprise gear implementing the server (open source nfs implementations are a joke and lack critical features), or basic maintenance like expanding storage or replication. The list goes on and on.

mjg59 7y ago
And yet Facebook appear to be successfully running a somewhat popular website while using NFS in production, which implies that the situation is somewhat less absolute than you claim.
- peterwwillis 7y ago
  Oh, you can definitely make money with a design from 2005 that uses NFS in production, 4 layers of caching, and enough servers that rebooting them constantly doesn't impact your service. I've worked at those companies. It was a bad idea then, it's a bad idea now.
  - richardwhiuk 7y ago
    Is there anything that's a good idea? Or is this just "everything has bugs and issues"?
    - deleted 7y ago
      [deleted]
    - notacoward 7y ago
      Seems like the latter. I'd guess peterwwillis has had some bad experiences with NFS. I've had some bad experience with some of the same so-called alternatives. Every time I see someone put a text blob into a database that needs to be in a file at point of use, I want to slap them. Ditto for everyone who tries to implement a hierarchical namespace on top of a flat blob store. Ditto for everyone who distributes a thousand copies of a rarely-changing file via scp or chef, then complains about update speed or inconsistency. NFS is not generally a good idea in high-volume high-node-count traffic paths, but it can be a less-bad solution in many other situations including many that are still part of production.
  - notacoward 7y ago
    That's not how it's done, and I know because I was in that group. Also, note that "production" at a company like Facebook includes a lot more than the direct web-content-serving path. I think I might agree that NFS doesn't make sense in that particular path, because it provides features and guarantees that nothing in that path needs and that means needless complexity, but there are plenty of other roles where NFS is a better fit than blob stores or copying files around.
    - peterwwillis 7y ago
      And there's plenty of other solutions that don't have all the problems (and overhead) NFS has. There are precious few actual good reasons to use NFS and they aren't customer facing products. Particularly, when you want a rootless thin terminal on an embedded system, or read-only transfer of non-sensitive files (say, for bootstrapping a machine from bootp, or sharing a corpus of data). For everything else (in production), NFS is a PITA, and most other networked filesystems should be avoided for similar reasons. Again: other solutions exist that provide the same functionality but do it without the inherent problems. Usually they are ignored because someone wanted to cut cost, or because legacy. Ask a storage engineer or infrastructure architect.