3 ms·

I was assuming a zero-copy manual buffer for read-ahead/write-behind. There is really no way mmap() could be faster since it is required to do more. The tradeof

I was assuming a zero-copy manual buffer for read-ahead/write-behind. There is really no way mmap() could be faster since it is required to do more. The tradeoff is that you don't have a linear address space (a non-issue for most apps) but you get much smarter scheduling and cache replacement behavior that results in higher effective throughput. You can't get there with madvise(); as the name suggests the kernel can and frequently does ignore very sensible requests to modify page cache behavior -- pathologically poor read-ahead behavior is what originally encouraged me to figure out kernel bypass. It is pretty easy to drive a gang of NVMe devices at their theoretical limits with a manual buffer if your access patterns are predictably simple, which is useful when slinging a lot of data around even if you aren't doing anything clever. The caveat is that even though the code to do it is simple, you still have to understand how Linux storage and APIs work. As you correctly observe, naive attempts that don't understand the fundamentals can be significantly worse than just mmap()-ing things, like never actually removing the page cache from your I/O path (common mistake).

vlovich123 5y ago
Sorry. I think I’m a little thick. How are you bypassing the page cache?
- jandrewrogers 5y ago
  Most common operating systems provide a facility to directly DMA disk pages into a user space buffer (and vice versa) without involving the page cache. On Linux, for example, this is what the O_DIRECT flag is for. It is similar to mmap() but with different rules; you lose the page table magic but gain explicit control the I/O behavior. There is nothing faster if you know how to wield it. Operating systems add this feature to support database engines but it isn't the only use case. Because a page cache is expected to support all possible processes, it can neither overfit simple access pattern cases, because then it would offer poor performance for common cases that are more complex, and it will never have enough context to offer excellent performance for complex access patterns because doing so would reduce performance for many common simple cases. Real page caches try to straddle both worlds; it is better to be mediocre for simple and complex cases than to be excellent at only simple or only complex cases. Bypassing the page cache allows processes to design their own page caches that are highly optimized for their specific requirements, should it be worthwhile to do so. A reason people might not want to do this is that the APIs are operating system specific.

vlovich123 5y ago

Sorry. I think I’m a little thick. How are you bypassing the page cache?

jandrewrogers 5y ago
Most common operating systems provide a facility to directly DMA disk pages into a user space buffer (and vice versa) without involving the page cache. On Linux, for example, this is what the O_DIRECT flag is for. It is similar to mmap() but with different rules; you lose the page table magic but gain explicit control the I/O behavior. There is nothing faster if you know how to wield it. Operating systems add this feature to support database engines but it isn't the only use case. Because a page cache is expected to support all possible processes, it can neither overfit simple access pattern cases, because then it would offer poor performance for common cases that are more complex, and it will never have enough context to offer excellent performance for complex access patterns because doing so would reduce performance for many common simple cases. Real page caches try to straddle both worlds; it is better to be mediocre for simple and complex cases than to be excellent at only simple or only complex cases. Bypassing the page cache allows processes to design their own page caches that are highly optimized for their specific requirements, should it be worthwhile to do so. A reason people might not want to do this is that the APIs are operating system specific.