3 ms·

We already have extremely high-bandwidth storage. Jens Axboe posts benchmarks on twitters using his io_uring test server, like this one[0] claiming 180 GB/s of

We already have extremely high-bandwidth storage. Jens Axboe posts benchmarks on twitters using his io_uring test server, like this one[0] claiming 180 GB/s of write throughput, which is fast enough to start blurring the lines with memory bandwidth. You can't use mmap() for large persistent data because CPU silicon doesn't support enough virtual memory, ignoring the other performance issues with current implementations. The elephant in the room is that using high-bandwidth storage well with minimal RAM requires different and much better scheduler design than currently exists in the vast majority of systems, including every OS I am familiar with. The higher the bandwidth and the larger the storage, the better your scheduling needs to be at latency hiding using techniques that don't involve a large cache. A lot of hardware tech, like Optane or HPC fabrics, was invented to avoid having to address schedulers being poor at latency hiding, which is essentially a (non-trivial) software problem. Most users of modern fast asynchronous I/O still tend to delegate scheduling, treating the I/O schedule and execution schedule as separate concerns, even when done entirely in user space. It is a missed opportunity. You do raise a valid issue: this requires a much more sophisticated design and implementation than most developers are comfortable with, which creates a lot of inertia behind doing things the classic way. This could all be abstracted away from the average developer in principle, something similar to a database kernel, but no one has built one yet. [0] https://x.com/axboe/status/1854635553775378458 https://x.com/axboe/status/1854635553775378458

kragen 2y ago
Thank you! This is extremely interesting indeed. My perhaps naïve thoughts on "extremely high bandwidth" are that the bus from the NAND Flash matrix to the Flash chip's internal RAM buffer is typically something like 4 kibibytes wide, and maybe you can read a page of the Flash into that buffer in hundreds of nanoseconds, though the chips I've looked into take tens of microseconds. (And if you can cut the latency down that much, the scheduler problems get much easier, too.) If those buffers are then accessible over the CPU's memory bus, maybe you can usefully transfer pages from the Flash into the buffers at many times the bandwidth of the CPU's memory bus, as long as you're only reading a small part of each page. As I understand it, current SSDs mostly only approach the bandwidth of SDRAM if you're reading entire pages. Back of the envelope: if you have 32 Flash chips in your system, and each one of them can read a 4096-byte page from its NAND matrix into a chip-internal RAM buffer every 100 ns, you'd have 1.3 terabytes per second of such bandwidth. However, this illustrates how demanding this kind of access would be to the hardware; basically it demands Flash as low-latency as SDRAM. You can definitely mmap() parts of devices or files that are larger than your CPU's virtual memory. mmap() does not require you to map the entire device or file. I agree that you need some abstraction layer that provides a simple, reliable interface with adequate performance.
- jandrewrogers 2y ago
  The I/O amplification from atomic 4KiB block size is definitely a thing. Even if you have the I/O bandwidth to burn this still represents a waste of useful RAM and memory bandwidth. There are a number of data layout and scheduling techniques that can significantly mitigate this. Latency is always going to have a speed-of-light issue, and storage is being moved physically further from the CPU with time; latency reductions in silicon are re-added by distance. Flash gets around that with extreme parallelism, which implies very deep pipelining from the execution scheduling side to fill all of the I/O slots necessary to saturate the bandwidth. This in turn creates a raft of second-order design problems on systems with extremely large and extremely parallel storage. Total memory requirements for just the execution state being scheduled, the data structures that provide data selectivity (so you can choose the optimal I/O to schedule), and landing buffers for parallel inflight I/O can easily overflow available RAM on large servers in plausible environments. And databases normally keep much more than just that in RAM. It is an interesting open architecture problem that has never been considered. You can't trivially patch existing architectures to make it work, it would need to look very different. We've taken it as axiomatic that certain data structures will always fit in RAM when dealing with large data, but extremely parallel, fast, and large storage is exposing that assumption in interesting ways. An architecture that effectively decouples memory requirements from large, fast storage would radically change how we design data intensive software, since there are a lot of design idioms and limitations today that primarily reflect RAM scaling issues.

kragen 2y ago

Thank you! This is extremely interesting indeed. My perhaps naïve thoughts on "extremely high bandwidth" are that the bus from the NAND Flash matrix to the Flash chip's internal RAM buffer is typically something like 4 kibibytes wide, and maybe you can read a page of the Flash into that buffer in hundreds of nanoseconds, though the chips I've looked into take tens of microseconds. (And if you can cut the latency down that much, the scheduler problems get much easier, too.) If those buffers are then accessible over the CPU's memory bus, maybe you can usefully transfer pages from the Flash into the buffers at many times the bandwidth of the CPU's memory bus, as long as you're only reading a small part of each page. As I understand it, current SSDs mostly only approach the bandwidth of SDRAM if you're reading entire pages. Back of the envelope: if you have 32 Flash chips in your system, and each one of them can read a 4096-byte page from its NAND matrix into a chip-internal RAM buffer every 100 ns, you'd have 1.3 terabytes per second of such bandwidth. However, this illustrates how demanding this kind of access would be to the hardware; basically it demands Flash as low-latency as SDRAM. You can definitely mmap() parts of devices or files that are larger than your CPU's virtual memory. mmap() does not require you to map the entire device or file. I agree that you need some abstraction layer that provides a simple, reliable interface with adequate performance.

jandrewrogers 2y ago
The I/O amplification from atomic 4KiB block size is definitely a thing. Even if you have the I/O bandwidth to burn this still represents a waste of useful RAM and memory bandwidth. There are a number of data layout and scheduling techniques that can significantly mitigate this. Latency is always going to have a speed-of-light issue, and storage is being moved physically further from the CPU with time; latency reductions in silicon are re-added by distance. Flash gets around that with extreme parallelism, which implies very deep pipelining from the execution scheduling side to fill all of the I/O slots necessary to saturate the bandwidth. This in turn creates a raft of second-order design problems on systems with extremely large and extremely parallel storage. Total memory requirements for just the execution state being scheduled, the data structures that provide data selectivity (so you can choose the optimal I/O to schedule), and landing buffers for parallel inflight I/O can easily overflow available RAM on large servers in plausible environments. And databases normally keep much more than just that in RAM. It is an interesting open architecture problem that has never been considered. You can't trivially patch existing architectures to make it work, it would need to look very different. We've taken it as axiomatic that certain data structures will always fit in RAM when dealing with large data, but extremely parallel, fast, and large storage is exposing that assumption in interesting ways. An architecture that effectively decouples memory requirements from large, fast storage would radically change how we design data intensive software, since there are a lot of design idioms and limitations today that primarily reflect RAM scaling issues.