6 ms·

Using mmap() is popular because it is easy despite its many sharp edges, but outside a limited number of access patterns and hardware setups it is often quite s

Using mmap() is popular because it is easy despite its many sharp edges, but outside a limited number of access patterns and hardware setups it is often quite slow because the page cache behavior is poor for many use cases. Manually buffering is often simpler, and with better performance, than trying to make the page cache perform well. Furthermore, mmap() doesn't work for large storage volumes generally, so you might find yourself doing your own buffering regardless. If you have a relatively simple access pattern, and most applications do, then you don't need to be a database kernel engineer to design a trivial manual buffering system that can significantly outperform mmap(). It is a handful of lines of code in most languages.

vlovich123 5y ago
Hmmm… this does not match my experience at all. Some user space buffering helps but that’s just because it’s easier to write your application logic (ie read 4 bytes, read 17 bytes, read 46kb). I have not heard the advice to layer another layer of buffer on top of if that. The reason is that it lets you write such code efficiently without paying the syscall overhead. These buffers are very stupid and io_uring or mmap (with a read ahead option) will perform better for larger files because there’s either no extra memory copy or the cost is hidden better. I’ve seen people struggle and fail to outperform the page cache. The only time I’ve beaten it is when it was caching the wrong thing (it was caching compressed and uncompressed pages via fuse so telling it to forget about the compressed pages after decompression eased memory pressure).
- jandrewrogers 5y ago
  I was assuming a zero-copy manual buffer for read-ahead/write-behind. There is really no way mmap() could be faster since it is required to do more. The tradeoff is that you don't have a linear address space (a non-issue for most apps) but you get much smarter scheduling and cache replacement behavior that results in higher effective throughput. You can't get there with madvise(); as the name suggests the kernel can and frequently does ignore very sensible requests to modify page cache behavior -- pathologically poor read-ahead behavior is what originally encouraged me to figure out kernel bypass. It is pretty easy to drive a gang of NVMe devices at their theoretical limits with a manual buffer if your access patterns are predictably simple, which is useful when slinging a lot of data around even if you aren't doing anything clever. The caveat is that even though the code to do it is simple, you still have to understand how Linux storage and APIs work. As you correctly observe, naive attempts that don't understand the fundamentals can be significantly worse than just mmap()-ing things, like never actually removing the page cache from your I/O path (common mistake).
  - vlovich123 5y ago
    Sorry. I think I’m a little thick. How are you bypassing the page cache?
    - jandrewrogers 5y ago
      Most common operating systems provide a facility to directly DMA disk pages into a user space buffer (and vice versa) without involving the page cache. On Linux, for example, this is what the O_DIRECT flag is for. It is similar to mmap() but with different rules; you lose the page table magic but gain explicit control the I/O behavior. There is nothing faster if you know how to wield it. Operating systems add this feature to support database engines but it isn't the only use case. Because a page cache is expected to support all possible processes, it can neither overfit simple access pattern cases, because then it would offer poor performance for common cases that are more complex, and it will never have enough context to offer excellent performance for complex access patterns because doing so would reduce performance for many common simple cases. Real page caches try to straddle both worlds; it is better to be mediocre for simple and complex cases than to be excellent at only simple or only complex cases. Bypassing the page cache allows processes to design their own page caches that are highly optimized for their specific requirements, should it be worthwhile to do so. A reason people might not want to do this is that the APIs are operating system specific.

vlovich123 5y ago

Hmmm… this does not match my experience at all. Some user space buffering helps but that’s just because it’s easier to write your application logic (ie read 4 bytes, read 17 bytes, read 46kb). I have not heard the advice to layer another layer of buffer on top of if that. The reason is that it lets you write such code efficiently without paying the syscall overhead. These buffers are very stupid and io_uring or mmap (with a read ahead option) will perform better for larger files because there’s either no extra memory copy or the cost is hidden better. I’ve seen people struggle and fail to outperform the page cache. The only time I’ve beaten it is when it was caching the wrong thing (it was caching compressed and uncompressed pages via fuse so telling it to forget about the compressed pages after decompression eased memory pressure).

jandrewrogers 5y ago
I was assuming a zero-copy manual buffer for read-ahead/write-behind. There is really no way mmap() could be faster since it is required to do more. The tradeoff is that you don't have a linear address space (a non-issue for most apps) but you get much smarter scheduling and cache replacement behavior that results in higher effective throughput. You can't get there with madvise(); as the name suggests the kernel can and frequently does ignore very sensible requests to modify page cache behavior -- pathologically poor read-ahead behavior is what originally encouraged me to figure out kernel bypass. It is pretty easy to drive a gang of NVMe devices at their theoretical limits with a manual buffer if your access patterns are predictably simple, which is useful when slinging a lot of data around even if you aren't doing anything clever. The caveat is that even though the code to do it is simple, you still have to understand how Linux storage and APIs work. As you correctly observe, naive attempts that don't understand the fundamentals can be significantly worse than just mmap()-ing things, like never actually removing the page cache from your I/O path (common mistake).
- vlovich123 5y ago
  Sorry. I think I’m a little thick. How are you bypassing the page cache?
  - jandrewrogers 5y ago
    Most common operating systems provide a facility to directly DMA disk pages into a user space buffer (and vice versa) without involving the page cache. On Linux, for example, this is what the O_DIRECT flag is for. It is similar to mmap() but with different rules; you lose the page table magic but gain explicit control the I/O behavior. There is nothing faster if you know how to wield it. Operating systems add this feature to support database engines but it isn't the only use case. Because a page cache is expected to support all possible processes, it can neither overfit simple access pattern cases, because then it would offer poor performance for common cases that are more complex, and it will never have enough context to offer excellent performance for complex access patterns because doing so would reduce performance for many common simple cases. Real page caches try to straddle both worlds; it is better to be mediocre for simple and complex cases than to be excellent at only simple or only complex cases. Bypassing the page cache allows processes to design their own page caches that are highly optimized for their specific requirements, should it be worthwhile to do so. A reason people might not want to do this is that the APIs are operating system specific.