3 ms·

Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s i

Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s. What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).

xtracto 23d ago
Could RAID0 help?
- adrian_b 23d ago
  When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0. RAID0 works fine only when all the interfaces have the same speed. If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD. In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs. One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
  - zamadatix 23d ago
    One approach I've seen used is: mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1 Which is effectively a way to get any positive integer m:n ratioed bandwidth distribution over any number of any sized drives.
    - adrian_b 23d ago
      This seems to be an acceptable solution. This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.

xtracto 23d ago

Could RAID0 help?

adrian_b 23d ago
When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0. RAID0 works fine only when all the interfaces have the same speed. If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD. In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs. One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
- zamadatix 23d ago
  One approach I've seen used is: mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1 Which is effectively a way to get any positive integer m:n ratioed bandwidth distribution over any number of any sized drives.
  - adrian_b 23d ago
    This seems to be an acceptable solution. This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.