5 ms·

Linux NFSv4.1 Performance Under a Microscope [pdf]

29 points by _delirium 12y ago

CyberDildonics 12y ago
If someone were making a network of workstations at a company that needed high performance from commodity hardware, what would be the best thing to use if they were starting from scratch?
- Nux 12y ago
  Local storage with SSDs and maybe some file sync tools.
- batbomb 12y ago
  There's not enough information to really answer this question. What is "high performance"? What is "commodity hardware"? How much storage do you actually need? What is the use of the data? Of course, if you mainly are using Hadoop you'll probably be playing around with HDFS. If you just have lots of Data and need to process it, maybe Ceph. If you need POSIX/NFS, you'll probably look at some sort of clustered NFS solution (a la GPFS). You've also got Gluster and Lustre. My general recommendation would probably be to examine Ceph first. Edit: I was assuming you meant some sort of networked storage system, I didn't realize it was for workstations. My answers are probably super overkill for what you need.
  - CyberDildonics 12y ago
    I'm wondering about workstations where everyone is accessing shared directories and gigabytes of data, so lots of IOPS and throughput to replace Netapp and not suffer 50MB/s. Basically my thought is that with combinations of SSDs, PCIe SSDs and things like FreeNAS (or some BSD+ZFS configuration) this shouldn't be an extravagant luxury.
    - voidlogic 12y ago
      You can now get 1 TB SSDs for ~$380, put 8 of these in RAID-10 (or 5 with HW RAID) shared over a 10 GBE NIC with NFS to workstations with 1 GBE / Jumbo frames. Nothing fancy, performance should be good.
      - CyberDildonics 12y ago
        That's great for a couple of computers but that isn't a full solution for a couple hundred people all hammering network storage. Also the original post is about NFS, so what do you do with the commodity setup, use NFS or something else?
        voidlogic 12y ago
        >but that isn't a full solution for a couple hundred people all hammering network storage. Come on, I have seen 8 10k SCSI drives/RAID10 in a Windows 2000 box be used by "a couple hundred people". It wasn't great, but this setup would blow that away, esp. if the hosting machine as a decent amount of RAM (32 GB+). Yes I was suggesting NFS + jumboframes.
    - suprjami 12y ago
      Sounds like you need a scale-out filesystem like Ceph or Gluster.
      - CyberDildonics 12y ago
        Thanks, I'll look in to those two.
- erglkjahlkh 12y ago
  That's too simple for specification. For instance: How much are you ready to pay for the performance in terms of usability, and what are the actual use cases? For instance NFS is pretty poor if your use case is serving shared network directories for workstations (not really great with complex ACLs), or if you require proper authentication for the resources (the Achilles' heel of NFS - where's my kerberos support?!?!!). You might be better off with CIFS. As the other comments already pointed out, there are several alternatives for several needs. Performance alone should not guide your selection.
  - uxcn 12y ago
    v4 (or v4.1) added a few things for increased security. You can probably use LDAP/Kerberos with it. In my experience, R/W throughput with CIFS is fairly bad.
    - erglkjahlkh 12y ago
      It seems I have some reading to do! I will check this out..
  - noselasd 12y ago
    We're using Kerberos to authenticate the NFSv4 shares, which seems to work fine. The usernames of the clients need to exist on the server though (but doesn't have to have the same uid as on the client) - you can probably set up a centralized user database with ldap, sssd, ipa or something like that - we havn't done that yet though.
  - xorcist 12y ago
    NFS has many problems but Kerberos support isn't one of them. It was designed from the start to be part of NFSv4, and is the most straightforward way to secure it.
- contingencies 12y ago
  You said 'network of workstations'. Then, you would match the application(s) to the hardware. For instance, in a 100% ground-up custom call center I used to run, we distributed diskless Linux to individual nodes and then had them pre-emptively cache portions of the NFS-mounted environment to tmpfs (ie. RAM) in order to preclude IO blocking. The point of the system (ie. the application) should come first, the architecture second (ie. in reality the architecture has to second-guess changes to the application, which generally come thick and fast). RAM is cheap. (Note: A non-workstation, general purpose computing cluster would have different architectural strategies.)

29 points by _delirium 12y ago

If someone were making a network of workstations at a company that needed high performance from commodity hardware, what would be the best thing to use if they were starting from scratch?

Nux 12y ago
Local storage with SSDs and maybe some file sync tools.
batbomb 12y ago
There's not enough information to really answer this question. What is "high performance"? What is "commodity hardware"? How much storage do you actually need? What is the use of the data? Of course, if you mainly are using Hadoop you'll probably be playing around with HDFS. If you just have lots of Data and need to process it, maybe Ceph. If you need POSIX/NFS, you'll probably look at some sort of clustered NFS solution (a la GPFS). You've also got Gluster and Lustre. My general recommendation would probably be to examine Ceph first. Edit: I was assuming you meant some sort of networked storage system, I didn't realize it was for workstations. My answers are probably super overkill for what you need.
- CyberDildonics 12y ago
  I'm wondering about workstations where everyone is accessing shared directories and gigabytes of data, so lots of IOPS and throughput to replace Netapp and not suffer 50MB/s. Basically my thought is that with combinations of SSDs, PCIe SSDs and things like FreeNAS (or some BSD+ZFS configuration) this shouldn't be an extravagant luxury.
  - voidlogic 12y ago
    You can now get 1 TB SSDs for ~$380, put 8 of these in RAID-10 (or 5 with HW RAID) shared over a 10 GBE NIC with NFS to workstations with 1 GBE / Jumbo frames. Nothing fancy, performance should be good.
    - CyberDildonics 12y ago
      That's great for a couple of computers but that isn't a full solution for a couple hundred people all hammering network storage. Also the original post is about NFS, so what do you do with the commodity setup, use NFS or something else?
      - voidlogic 12y ago
        >but that isn't a full solution for a couple hundred people all hammering network storage. Come on, I have seen 8 10k SCSI drives/RAID10 in a Windows 2000 box be used by "a couple hundred people". It wasn't great, but this setup would blow that away, esp. if the hosting machine as a decent amount of RAM (32 GB+). Yes I was suggesting NFS + jumboframes.
  - suprjami 12y ago
    Sounds like you need a scale-out filesystem like Ceph or Gluster.
    - CyberDildonics 12y ago
      Thanks, I'll look in to those two.
erglkjahlkh 12y ago
That's too simple for specification. For instance: How much are you ready to pay for the performance in terms of usability, and what are the actual use cases? For instance NFS is pretty poor if your use case is serving shared network directories for workstations (not really great with complex ACLs), or if you require proper authentication for the resources (the Achilles' heel of NFS - where's my kerberos support?!?!!). You might be better off with CIFS. As the other comments already pointed out, there are several alternatives for several needs. Performance alone should not guide your selection.
- uxcn 12y ago
  v4 (or v4.1) added a few things for increased security. You can probably use LDAP/Kerberos with it. In my experience, R/W throughput with CIFS is fairly bad.
  - erglkjahlkh 12y ago
    It seems I have some reading to do! I will check this out..
- noselasd 12y ago
  We're using Kerberos to authenticate the NFSv4 shares, which seems to work fine. The usernames of the clients need to exist on the server though (but doesn't have to have the same uid as on the client) - you can probably set up a centralized user database with ldap, sssd, ipa or something like that - we havn't done that yet though.
- xorcist 12y ago
  NFS has many problems but Kerberos support isn't one of them. It was designed from the start to be part of NFSv4, and is the most straightforward way to secure it.
contingencies 12y ago
You said 'network of workstations'. Then, you would match the application(s) to the hardware. For instance, in a 100% ground-up custom call center I used to run, we distributed diskless Linux to individual nodes and then had them pre-emptively cache portions of the NFS-mounted environment to tmpfs (ie. RAM) in order to preclude IO blocking. The point of the system (ie. the application) should come first, the architecture second (ie. in reality the architecture has to second-guess changes to the application, which generally come thick and fast). RAM is cheap. (Note: A non-workstation, general purpose computing cluster would have different architectural strategies.)