> I am less sure about madvise. It seems to me default heuristics should work fine for this case, and as you said, system calls are expensive. System calls are

> I am less sure about madvise. It seems to me default heuristics should work fine for this case, and as you said, system calls are expensive. System calls are expensive but so are page faults or having to access the disk. If you can avoid page faults by using madvise to prefetch from disk to memory, it should be worth it. In particular, the first run with cold caches should be faster. However, the operating system may be smart and realize that we're doing a sequential access and may speculatively read ahead and madvise calls would be time wasted. The same happens with CPU caches too, the CPU internal prefetcher is pretty good in recognizing a sequential access and grabbing the next cache line in advance. A few naively placed __builtin_prefetches doesn't seem to help here (I just tried this out). Prefetching hints work a lot better in non-sequential access patterns (linked lists, etc).