3 ms·
There are ways to be more clever than just mmap() + madvise() to improve performance in cases that might block (e.g. mincore() and some other system interfaces)
by jandrewrogers 14y ago
There are ways to be more clever than just mmap() + madvise() to improve performance in cases that might block (e.g. mincore() and some other system interfaces) but the further down that road you go the less portable and more complicated the implementation becomes.
In short, mmap() and friends were designed for a world where context-switching (e.g. multithread or multiprocess) is a great idea. Unfortunately, context-switching has become extraordinarily expensive for a lot of server software which makes multithreading a less palatable option.
These days, you want (1) native async I/O, (2) strict control of cache replacement behavior, (3) strict control of I/O scheduling, (4) minimal context-switching, and (5) memory locality control. On Linux, this means DIRECT_IO, io_submit, managing your own physical cache RAM, and locking one thread to every core (ignoring hyperthreads for the moment). This is more complicated to implement because there is not a simple, portable interface like mmap() widely available but it is also much more efficient and performant when done correctly. To make matters worse, some useful interfaces (like io_submit) are poorly documented.
But yes, if you build your own memory and swapping subsystem directly on top of the native interfaces, and use threads as an abstraction of a core rather than a swappable context, you can build very efficient server engines that stall minimally on I/O. (Note: even using io_submit and IO_DIRECT for async disk access, there are conditions that can cause blocking. They are just much rarer and easier to manage than mmap().)