3 ms·

Thanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works. If you have any co

Thanks! Yeah I've seen both general advice but since I'm not incredibly familiar with the API I've had a hard time creating code that works. If you have any code examples of either of those options I'd love to see them! Edit: Also, I'm curious: what are the downsides to doing the mmap route?

loeg 4y ago
Downsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again). You can mitigate this somewhat with explicit MAP_POPULATE and mlock.
- Sesse__ 4y ago
  Also, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.
  - loeg 4y ago
    > more memory use for very large files (the page tables are not free), They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore). If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down. > no way of handling errors. Yeah, this aspect is really significant. Thanks for mentioning it.
    - Sesse__ 4y ago
      0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.
      - loeg 4y ago
        Oh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.

loeg 4y ago

Downsides - unpredictable latency for IO when the parser crosses a page boundary and encounters an unbacked page. Unpredictable IO issue size (might be smaller IOs than ideal). Unpredictable caching behavior (OS may drop the pages before the application is done with them, only to have to reread again). You can mitigate this somewhat with explicit MAP_POPULATE and mlock.

Sesse__ 4y ago
Also, generally higher overhead, more memory use for very large files (the page tables are not free), and no way of handling errors.
- loeg 4y ago
  > more memory use for very large files (the page tables are not free), They're... kinda free (in terms of capacity). A PTE or PDE is 64 bits per page. Even with 4kB pages, this is 8/4096 = 0.2% overhead. Churning the page tables and TLB as you walk the file is expensive, or at least historically was expensive (used to be that removing an entry required flushing the entire TLB; I don't think that's true anymore). If you can use MAP_HUGE_2MB, this drops off to 0.00038% and the TLB impact goes way down. > no way of handling errors. Yeah, this aspect is really significant. Thanks for mentioning it.
  - Sesse__ 4y ago
    0.2% really isn't free. If you have a data set of 10 TB (one small-ish HDD right now, or one large SSD), that's ~20 GiB in page tables alone. If you can use 2MB pages, then sure, but I would assume that also limits your page-in granularity.
    - loeg 4y ago
      Oh, you're right. I wasn't remembering we needed PTEs to cover the data not paged in as well as the data in RAM. Sorry.