6 ms·
That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.
by Aissen 5y ago
That's 125 ns per IO, on a single CPU core, with two devices. Mind boggling.
- megous 5y agoIO is 512B sectors here? Or 4KiB? EDIT: 512, looking at the screenshot..
- guerrilla 5y agoFaster than DRAM from the 90's.
- spicybright 5y agoI'd hope so lol
- formerly_proven 5y agoThe access time has actually not changed a lot, it's just that transferring the smallest read a contemporary processor would issue - one cache line (~64 bytes) - at somewhere between 500-1000 MB/s takes around 100 ns already. That transfer-related latency has been reduced drastically.
- guerrilla 5y agoDidn't /RAS timing drop from like 120ns to 10ns too? That's what I was thinking about. Then DDR, and that bus size is also 16 times wider.
- formerly_proven 5y agoWasn't tRAS always around 2-3x tCL?
- terafo 5y agoModern SSDs generally have more bandwidth than DDR2.
- devit 5y agoAt around 4 GHz and 2 IPC, it's about 1000 CPU instructions per I/O. Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. Without the direct mapped ring buffer and with a filesystem, you need a kernel to translate from uring to the hardware ring buffer, and here it still seems around 10x inefficient as around 100 instructions should be enough to do the translation (assuming pages already mapped in the IOMMU, that you have the file block map in cache, and that the whole system is architected to maximize the efficiency of this operation).
- throwawaylinux 5y ago> Assuming you have an hardware DMA ring buffer for I/O that is directly mapped in userspace, the only thing that is really needed is to write the operation type, size, disk position and memory position to the ring buffer, update the buffer position and check for flush, doable in around 8 CISC instructions (plus the slowpath), so around 100x inefficient. This type of thing is not measured in instructions anymore, it's measured in off-board operations and latencies. Cache misses, latencies to poke MMIO and get back interrupts if necessary, DMA transfer to complete, device access time. In this case it seems the hardware is theoretically capable of about 12M so the core mostly be just waiting for that.
- Flow 5y agoWill DMA from external devices pass L3 or some other CPU cache? Or are all accesses to the freshy DMAed-to memory be cold
- anarazel 5y agoIt's worth noting that there's some batching involved, so it's not 125ns for processing one io, then again 125ns for the next. Both combining some of the work, and superscalar execution is necessary to reach this high numbers...
- benlwalker 5y agoTo expand on the batching details, I haven't seen exactly what he's doing here, but historically the numbers quoted are from benchmarks that submit something like 128 total queue depth in alternating batches of 64. The disks hit max performance with a much lower queue depth than 64. So it's very unrealistic perfect batching that you'll never see in a real application. The workload is only a useful tool for optimizing the code, which is exactly what he's using it for.
- wtallis 5y agoThe Intel Optane drives he's testing with hit max performance at pretty low queue depths, but flash-based SSDs with similar throughput will require those high queue depths.
- anarazel 5y agoI think you need batching for the nvme doorbell alone at anything close to these kinds of rates... For some workloads it's not that hard to have that deep queues. What's harder is to know when to use them and when not. There's really not enough information available to make any of this self-tuning.