3 ms·
Modern AMD Zen2/3 has more PCIe bandwidth than DRAM bandwidth. In other words, you can't saturate your flash if you buffer reads through DRAM, because the buffe
by namibj 4y ago
Modern AMD Zen2/3 has more PCIe bandwidth than DRAM bandwidth.
In other words, you can't saturate your flash if you buffer reads through DRAM, because the buffering costs you one write followed a bit later by one read.
mmap hits limits far earlier due to the kernel evicting pages with only a single thread and much of the process needing global-ish locks.
Use zoned storage and raw NVMe, with a fallback to io_uring. You need a userspace page cache of some sort. Maybe randomly sample the stack of pages you traversed to get to the page you're currently looking at, and bump them in the LRU.
Feel free to default to stream latency-insensitive table scan operations without even caching them to not pollute cache.
- dataangel 4y agoHow do you avoid buffering reads through DRAM and also avoid using mmap? Like even io_uring has to get data off the device (which is probably memory mapped) and into DRAM, so maybe I'm just not understanding what you mean.
- gpderetta 4y agoSome PCI devices can DMA directly into/from the CPU cache bypassing memory. See intel direct IO for example.
- namibj 4y agoYou can for example use a controller memory buffer with NVMe and just access the data over mapped PCIe, or you get it to DMA into L3 cache and use/overwrite it before it gets it's turn at the write-back part of the DRAM memory controller.