4 ms·
How is that different to virtual memory?
by hermitcrab 6y ago
How is that different to virtual memory?
- dTal 6y agoMemory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.
- wtallis 6y agoYou've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.
- hermitcrab 6y agoThanks for clearing that up!
- nijave 6y agoIn memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access