28 ms·

The diagram is a bit misleading. There is no system call (switching to the kernel) with read or write hits, assuming the memory is already mapped, has a page ta

The diagram is a bit misleading. There is no system call (switching to the kernel) with read or write hits, assuming the memory is already mapped, has a page table entry and is resident. You only incur switches when performing the mapping and due to various page fault modes (lazy mapping, non-resident data, copy-on-write needed). > This usually happens when the ratio of storage size to RAM size is significantly higher than 1:1. Every page that is brought into cache causes another page to be evicted. While true one can optimize these things by unmapping larger ranges in bulk (to reduce cache pressure) or prefetching them (to reduce blocking) with madvise, allowing the kernel doing the loading asynchronously while you're accessing the previously prefetched ones. If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads.

arielweisberg 9y ago
I noticed that as well. Maybe it's assuming a CPU cache and TLB miss on read which means going to the kernel. Also on write if it's a new page you are going to have to fault once for every page which depending on write size is like having to go to the kernel for every write. It can actually be good for performance in some dimensions to pre-touch pages in a loop. I can't speak for Avi though. I suspect he is writing from the perspective of a database implementor. For a database the page cache is not predictable in ways you want it to be. For writes the issue is IO scheduling, when flushing occurs, and how much is flushed at a time and at what rate. For reads if a page is not in memory you have to block a thread to read it. If you are thread per core this means you either have a thread pool on the side to do IO or you block an event loop preventing other work from making progress. Even having the thread pool on the side is problematic because checking whether a page is in the cache is not cheap either and requires a system call which is kind of self defeating.
- the8472 9y ago
  > Maybe it's assuming a CPU cache and TLB miss on read which means going to the kernel. My understanding is that those normally don't incur faults that would switch to the kernel context. They're just slow due to several memory lookups that have to be done to satisfy the primary memory access.
  - arielweisberg 9y ago
    Oh interesting. TIL.
    - bonzini 9y ago
      Actually it depends on the processor. Nowadays it's rare, but in the past there used to be software TLBs where a TLB miss did trap into the kernel, and there was no radix tree-like page table structure in the definition of the architecture. SPARC and MIPS for example worked like this, and server POWER chips had a hashed page table that many OSes populated lazily, effectively treating it like a (more efficient) software TLB.
      - namibj 9y ago
        Well, MIPS is still in widespread use in wifi hardware, and POWER is being resurrected for a scale-up way to deal with a growing database, considering the current generation scales higher at about a similar pricepoint than a 4-socket Intel machine. So I am not sure what the past tense is supposed to mean, if it denotes the architectures to no longer compromise a significant part of world-wide (say DRYSTONE) MIPS, or their lack of use where they are good at (false).
        bonzini 9y ago
        I am not sure about MIPS, but other processors have moved away from either software TLBs or hashed page tables (POWER9 supports tree page tables). Hence the past tense.
  - richardwhiuk 9y ago
    Correct for x86/x64. TLB misses require the CPU to lookup the location in the current active page table which means another memory lookup. I'm not sure how the cache hierarchy works for page table lookups following a TLB miss, but worst case, it's equivalent to an additional L3 cache miss.
seastarer 9y ago
Right, the mmap read hit is incorrect, I'll fix it up.
- seastarer 9y ago
  I see what happened, the read/write diagram got placed under mmap.
mtanski 9y ago
In really new linux there is a preadv2 sysscall that was merged. preadv2 supports a flag RWF_NOWAIT to tell the kernel to not block if a read requires uncached data. This lets you play around with policy for blocking reads in user space. Examples: try to make progress on partially read data & queue up the rest on another thread; try performing reads from your network thread and if unavailable queue up the blocking read onto another disk thread & serve a different request. There's no RWF_NOWAIT support for write in pwritev2. But it's technically possible to implement it and if your write will cause a writeout to disk return EWOULDBLOCK. LWN description: https://lwn.net/Articles/612483/ https://lwn.net/Articles/612483/ LWN summary of my talk: https://lwn.net/Articles/636967/ https://lwn.net/Articles/636967/ Final patch set by Christoph: https://lwn.net/Articles/731700/ https://lwn.net/Articles/731700/ Disclaimer: I'm the author of the preadv2/pwritev2 syscalls. And the original author of the support for RWF_NOWAIT, which Christoph Hellwig took over. So feel free to consider this to be self congratulatory. Correction: It looks like there's a patch set floating around for adding support for RWF_NOWAIT in pwritev2 as well. https://patchwork.kernel.org/patch/9787271/ https://patchwork.kernel.org/patch/9787271/ (comment from Christoph)
- colanderman 9y ago
  > try performing reads from your network thread and if unavailable How does this differ from recv(..., MSG_DONTWAIT)?
  - tmzt 9y ago
    I think he means perform a disk read in response to a network request, and choose to respond to a different queued/pooled request if the data is not available to complete it. I would assume the intent is similar but applied to a different type/source of IO.
  - mtanski 9y ago
    This was my original model when I came up with the idea; in fact, my v1 hack used recv and MSG_DONTWAIT. For a number of a reasons the kernel community did not want to overload that interface and that flag. Besides some technical reasons, the big reason was an "impedance mismatch" of the recv API which was working with stream data. In that context you're depending on another party send data. Also, you can wait on this to happen using another API (select et. al) so the buffer will be filled not by your actions. On the other hand preadv2(..., RWF_NOWAIT) is not going to trigger any more read in and theres no wait to wait on the data. Although the previous statement is not a 100% true... preadv2 may or may not trigger readahead (if it's enabled). Here's the whole thread about it... if you're interested in the history of how this came to be: https://lkml.org/lkml/2014/7/24/787 https://lkml.org/lkml/2014/7/24/787
colanderman 9y ago
> If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads. The same is true of plain read/write, barring extra data copies to/from the kernel. The only reason to ever use AIO is if you're doing direct I/O, in which case mmap doesn't apply.
- srcmap 9y ago
  "If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads." I have seen Windows File Explorer lock up over and over again for the same assumption. They assume the dir scan is fast operation. A slow, bad USB drive, NFS, Samba mount, or trying to scan directory with thousands of files will lockup the GUI for long period of time. Dragging a file over slow/busy system icon with SMB mount will just spin / hang the whole GUI - not a very good UX.
  - deleted 9y ago
    [deleted]
seastarer 9y ago
The diagrams are now correct (the read/write diagram was placed under the mmap section by mistake).
seastarer 9y ago
> If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads. This may work for sequential reads, but not for random reads.
- the8472 9y ago
  Not just sequential reads. Any reads where you know the pattern and can advise in advance.

arielweisberg 9y ago

I noticed that as well. Maybe it's assuming a CPU cache and TLB miss on read which means going to the kernel. Also on write if it's a new page you are going to have to fault once for every page which depending on write size is like having to go to the kernel for every write. It can actually be good for performance in some dimensions to pre-touch pages in a loop. I can't speak for Avi though. I suspect he is writing from the perspective of a database implementor. For a database the page cache is not predictable in ways you want it to be. For writes the issue is IO scheduling, when flushing occurs, and how much is flushed at a time and at what rate. For reads if a page is not in memory you have to block a thread to read it. If you are thread per core this means you either have a thread pool on the side to do IO or you block an event loop preventing other work from making progress. Even having the thread pool on the side is problematic because checking whether a page is in the cache is not cheap either and requires a system call which is kind of self defeating.

the8472 9y ago
> Maybe it's assuming a CPU cache and TLB miss on read which means going to the kernel. My understanding is that those normally don't incur faults that would switch to the kernel context. They're just slow due to several memory lookups that have to be done to satisfy the primary memory access.
- arielweisberg 9y ago
  Oh interesting. TIL.
  - bonzini 9y ago
    Actually it depends on the processor. Nowadays it's rare, but in the past there used to be software TLBs where a TLB miss did trap into the kernel, and there was no radix tree-like page table structure in the definition of the architecture. SPARC and MIPS for example worked like this, and server POWER chips had a hashed page table that many OSes populated lazily, effectively treating it like a (more efficient) software TLB.
    - namibj 9y ago
      Well, MIPS is still in widespread use in wifi hardware, and POWER is being resurrected for a scale-up way to deal with a growing database, considering the current generation scales higher at about a similar pricepoint than a 4-socket Intel machine. So I am not sure what the past tense is supposed to mean, if it denotes the architectures to no longer compromise a significant part of world-wide (say DRYSTONE) MIPS, or their lack of use where they are good at (false).
      - bonzini 9y ago
        I am not sure about MIPS, but other processors have moved away from either software TLBs or hashed page tables (POWER9 supports tree page tables). Hence the past tense.
- richardwhiuk 9y ago
  Correct for x86/x64. TLB misses require the CPU to lookup the location in the current active page table which means another memory lookup. I'm not sure how the cache hierarchy works for page table lookups following a TLB miss, but worst case, it's equivalent to an additional L3 cache miss.

seastarer 9y ago

Right, the mmap read hit is incorrect, I'll fix it up.

seastarer 9y ago
I see what happened, the read/write diagram got placed under mmap.

mtanski 9y ago

In really new linux there is a preadv2 sysscall that was merged. preadv2 supports a flag RWF_NOWAIT to tell the kernel to not block if a read requires uncached data. This lets you play around with policy for blocking reads in user space. Examples: try to make progress on partially read data & queue up the rest on another thread; try performing reads from your network thread and if unavailable queue up the blocking read onto another disk thread & serve a different request. There's no RWF_NOWAIT support for write in pwritev2. But it's technically possible to implement it and if your write will cause a writeout to disk return EWOULDBLOCK. LWN description: https://lwn.net/Articles/612483/ https://lwn.net/Articles/612483/ LWN summary of my talk: https://lwn.net/Articles/636967/ https://lwn.net/Articles/636967/ Final patch set by Christoph: https://lwn.net/Articles/731700/ https://lwn.net/Articles/731700/ Disclaimer: I'm the author of the preadv2/pwritev2 syscalls. And the original author of the support for RWF_NOWAIT, which Christoph Hellwig took over. So feel free to consider this to be self congratulatory. Correction: It looks like there's a patch set floating around for adding support for RWF_NOWAIT in pwritev2 as well. https://patchwork.kernel.org/patch/9787271/ https://patchwork.kernel.org/patch/9787271/ (comment from Christoph)

colanderman 9y ago
> try performing reads from your network thread and if unavailable How does this differ from recv(..., MSG_DONTWAIT)?
- tmzt 9y ago
  I think he means perform a disk read in response to a network request, and choose to respond to a different queued/pooled request if the data is not available to complete it. I would assume the intent is similar but applied to a different type/source of IO.
- mtanski 9y ago
  This was my original model when I came up with the idea; in fact, my v1 hack used recv and MSG_DONTWAIT. For a number of a reasons the kernel community did not want to overload that interface and that flag. Besides some technical reasons, the big reason was an "impedance mismatch" of the recv API which was working with stream data. In that context you're depending on another party send data. Also, you can wait on this to happen using another API (select et. al) so the buffer will be filled not by your actions. On the other hand preadv2(..., RWF_NOWAIT) is not going to trigger any more read in and theres no wait to wait on the data. Although the previous statement is not a 100% true... preadv2 may or may not trigger readahead (if it's enabled). Here's the whole thread about it... if you're interested in the history of how this came to be: https://lkml.org/lkml/2014/7/24/787 https://lkml.org/lkml/2014/7/24/787

colanderman 9y ago

> If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads. The same is true of plain read/write, barring extra data copies to/from the kernel. The only reason to ever use AIO is if you're doing direct I/O, in which case mmap doesn't apply.

srcmap 9y ago
"If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads." I have seen Windows File Explorer lock up over and over again for the same assumption. They assume the dir scan is fast operation. A slow, bad USB drive, NFS, Samba mount, or trying to scan directory with thousands of files will lockup the GUI for long period of time. Dragging a file over slow/busy system icon with SMB mount will just spin / hang the whole GUI - not a very good UX.
- deleted 9y ago
  [deleted]

seastarer 9y ago

The diagrams are now correct (the read/write diagram was placed under the mmap section by mistake).

seastarer 9y ago

> If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads. This may work for sequential reads, but not for random reads.

the8472 9y ago
Not just sequential reads. Any reads where you know the pattern and can advise in advance.