4 ms·

A non-obvious drawback of this is you give up multi-threaded compression (maybe fixed in a later release). You also need to use the same memory size when decomp

• terrelln 9y ago
  We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.
  • ghusbands 9y ago
    I understand this to mean that you believe the pages in the long range aren't accessed frequently. Otherwise, mmap doesn't reduce memory usage so much as just hide it. Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
    • terrelln 9y ago
      Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed. The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.