3 ms·

the feature you're looking for is called opening a memory mapped file. I've seen tremendous performance improvements, but it's a really underused feature for so

• wtallis 8y ago
  Does Windows have any analog to madvise()? It seems like that's probably necessary to making things work well when you're mapping huge data sets, unless you're sure they reside on really fast storage.
  • PeCaN 8y ago
    Windows 8 and later has PrefetchVirtualMemory[1]. Before that your best bet is to get a little hacky and try reading the file asynchronously to get it in the cache (Windows caches files pretty aggressively, so this more or less works). [1] https://msdn.microsoft.com/en-us/library/windows/desktop/hh780543(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/hh7...
    • senozhatsky 8y ago
      Seems that VirtualAlloc() [1] is a little bit closer to madvise(), but still is pretty far. madvise() is quite powerful. For instance, take a look at MADV_FREE. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa366887(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/aa3... -ss
      • PeCaN 8y ago
        I figured the use-case here was MADV_WILLNEED, which is what PrefetchVirtualMemory is for. VirtualAlloc flags would be for things like reserving virtual pages without committing them. There's no single API on Windows that does the myriad of things that madvise does.
• ddorian43 8y ago
  Yep, see lmdb read performance. Or many search engines doing mmap.
• bluejekyll 8y ago
  > it's a really underused feature for some reason My suspicion is that it’s mainly because memory mapping files, while safe for reading, have a whole bunch of edge cases where the file can accidentally be corrupted.