3 ms·
Dumb question, but wouldn't memory mapping or lazy/streaming processing (e.g. SAX for parsing large XML documents, Java Streams for handling large amounts of "t
by Sahonon 7y ago
Dumb question, but wouldn't memory mapping or lazy/streaming processing (e.g. SAX for parsing large XML documents, Java Streams for handling large amounts of "things", memory mapped files for anything you don't need intermediate representations for) resolve this problem as well?
- itamarst 7y agoSee above about streaming (chunking is basically the same thing). mmaping is a form of uncontrolled chunking, where chunk size and location is determined by operating system's filesytem caching policy, readahead heuristics, and the like. As a result, it can sometimes have much worse performance than an explicit chunking strategy, especially if you just treat it as magic. Or to put it another way, mmaping is helpful but you still need to understand why it might help and how to use it.
- faizshah 7y ago> it can sometimes have much worse performance than an explicit chunking strategy Can you expand on this? I haven’t heard this before. Edit: Found a comprehensive discussion on this https://stackoverflow.com/questions/45972/mmap-vs-reading-blocks https://stackoverflow.com/questions/45972/mmap-vs-reading-bl...