3 ms·
Yes, this paper did not compare small file performance in HopsFS with EMRFS/S3. But HopsFS on S3 also supports storing small files on NVMe disks in the metadata
by jamesblonde 6y ago
Yes, this paper did not compare small file performance in HopsFS with EMRFS/S3. But HopsFS on S3 also supports storing small files on NVMe disks in the metadata layer. Previously, we have shown that we can get up to 66X higher throughput for writes for small files versus HDFS (peer reviewed, of course) -
https://www.logicalclocks.com/blog/millions-and-millions-of-files-for-deep-learning-with-hopsfs https://www.logicalclocks.com/blog/millions-and-millions-of-...