3 ms·

How did you get around the terrible performance of EFS. Are you not read/writing lots of small files?

haggy 8y ago
Nobody should be writing many small files to EFS/S3/etc. It's very bad at that.
brazzledazzle 8y ago
NFS in general seems to struggle with that scenario. Do you get better performance with your own NFS server with that type of workload?
- chucky_z 8y ago
  Unrelated to EFS, but I found Gluster to have good performance for this use-case vs. NFS/CIFS.
- X-Istence 8y ago
  A NetApp filer on 40 Gbit/sec will do NFS plenty fast for running VMs directly to/from the system using ESXi for example. EFS is not even in the same ballpark in terms of speed :/
  - brazzledazzle 8y ago
    Right, I’ve had that setup in the past and I wouldn’t disagree with that but I’ve also seen a netapp choke on lots of tiny files over NFS.
shoo 8y ago
Good question. The application had grown and ran in production over a number of years using NFS for storage before it was moved into AWS, so this probably naturally steered devs away from trying to use the shared file storage as a high performance cache (eg they might try it, and find it to be slow, and figure out a different way of doing whatever they needed to do, probably using the db). Our usage pattern for NFS did not generally involve needing to read or write many small tiny files with any degree of performance. E.g. we might need to do reads/writes of dozens of files, each of a few mb in a process, with the overall running time of the process being largely governed by our DB performance / patches of poorly written algorithmic code running in a slow scripting language. Amdahl's law - for us making the NFS access go infinitely fast or 5x slower hardly changes the overall total process execution time. I can certainly appreciate that EFS is an abysmally slow for dealing with large numbers of reads/ writes of tiny files, in another part of the system one of my colleagues set up a read only cache of a large number of tiny files in EFS, due to the per-file write latency it took around a week to load in < 100gb of data.

Nobody should be writing many small files to EFS/S3/etc. It's very bad at that.

NFS in general seems to struggle with that scenario. Do you get better performance with your own NFS server with that type of workload?

chucky_z 8y ago
Unrelated to EFS, but I found Gluster to have good performance for this use-case vs. NFS/CIFS.
X-Istence 8y ago
A NetApp filer on 40 Gbit/sec will do NFS plenty fast for running VMs directly to/from the system using ESXi for example. EFS is not even in the same ballpark in terms of speed :/
- brazzledazzle 8y ago
  Right, I’ve had that setup in the past and I wouldn’t disagree with that but I’ve also seen a netapp choke on lots of tiny files over NFS.

shoo 8y ago

Good question. The application had grown and ran in production over a number of years using NFS for storage before it was moved into AWS, so this probably naturally steered devs away from trying to use the shared file storage as a high performance cache (eg they might try it, and find it to be slow, and figure out a different way of doing whatever they needed to do, probably using the db). Our usage pattern for NFS did not generally involve needing to read or write many small tiny files with any degree of performance. E.g. we might need to do reads/writes of dozens of files, each of a few mb in a process, with the overall running time of the process being largely governed by our DB performance / patches of poorly written algorithmic code running in a slow scripting language. Amdahl's law - for us making the NFS access go infinitely fast or 5x slower hardly changes the overall total process execution time. I can certainly appreciate that EFS is an abysmally slow for dealing with large numbers of reads/ writes of tiny files, in another part of the system one of my colleagues set up a read only cache of a large number of tiny files in EFS, due to the per-file write latency it took around a week to load in < 100gb of data.