5 ms·

Zstd recently added a long range mode that can find matches up to 2 GB in the past using a specialized algorithm [1]. It can be enabled on the command line with

Zstd recently added a long range mode that can find matches up to 2 GB in the past using a specialized algorithm [1]. It can be enabled on the command line with `zstd --long` for a 128 MB window, or `--long=windowLog` for a `2^windowLog` byte window. [1] https://github.com/facebook/zstd/releases/tag/v1.3.2 https://github.com/facebook/zstd/releases/tag/v1.3.2

Willson50 9y ago
A non-obvious drawback of this is you give up multi-threaded compression (maybe fixed in a later release). You also need to use the same memory size when decompressing.
- terrelln 9y ago
  We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.
  - ghusbands 9y ago
    I understand this to mean that you believe the pages in the long range aren't accessed frequently. Otherwise, mmap doesn't reduce memory usage so much as just hide it. Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
    - terrelln 9y ago
      Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed. The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.
exikyut 9y ago
I'm not sure if you'll see this, but I have a question. I have a situation where I need to implement in-place compression with perfect rollback in case of failure. The first approach I thought of was feeding LZMA input until I had X amount of output, then finalizing and starting over, but then I started wondering if I could simply periodically serialize the LZMA state machine data every 30 seconds or so. I haven't investigated either approach, but this is one of my next projects I'm working on, so I'll be figuring it out either way. I was wondering, does Zstd provide any ability to serialize/restore and continue where it left off? (Context: Disk compression. Due to long-term factors beyond my control I've had severe disk space issues for over a decade (with frequently only tens of MBs free). Complex story ending in "cannot work". I recently realized that some of my disks contained data I didn't immediately need, and began wondering about ways I might be able to compress this data out "of the way" so I could carefully partition the remaining space. This would need to be done in-place as I would have nowhere with enough space to hold the compressed data, at least not to begin with.)
- mkesper 9y ago
  Lending/buying an empty (external) drive for starting seems to be much simpler and safer.
  - exikyut 9y ago
    That would be a highly preferable option. The last time someone gave me an old empty 320GB HDD they weren't using, one of my own disks started clicking about a week later and I was able to save everything on it. I still shake my head at that perfect timing. Of course, this meant I lost all the additional free space, haha. One step forward, two steps back...

Willson50 9y ago

A non-obvious drawback of this is you give up multi-threaded compression (maybe fixed in a later release). You also need to use the same memory size when decompressing.

terrelln 9y ago
We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.
- ghusbands 9y ago
  I understand this to mean that you believe the pages in the long range aren't accessed frequently. Otherwise, mmap doesn't reduce memory usage so much as just hide it. Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
  - terrelln 9y ago
    Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed. The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.

exikyut 9y ago

I'm not sure if you'll see this, but I have a question. I have a situation where I need to implement in-place compression with perfect rollback in case of failure. The first approach I thought of was feeding LZMA input until I had X amount of output, then finalizing and starting over, but then I started wondering if I could simply periodically serialize the LZMA state machine data every 30 seconds or so. I haven't investigated either approach, but this is one of my next projects I'm working on, so I'll be figuring it out either way. I was wondering, does Zstd provide any ability to serialize/restore and continue where it left off? (Context: Disk compression. Due to long-term factors beyond my control I've had severe disk space issues for over a decade (with frequently only tens of MBs free). Complex story ending in "cannot work". I recently realized that some of my disks contained data I didn't immediately need, and began wondering about ways I might be able to compress this data out "of the way" so I could carefully partition the remaining space. This would need to be done in-place as I would have nowhere with enough space to hold the compressed data, at least not to begin with.)

mkesper 9y ago
Lending/buying an empty (external) drive for starting seems to be much simpler and safer.
- exikyut 9y ago
  That would be a highly preferable option. The last time someone gave me an old empty 320GB HDD they weren't using, one of my own disks started clicking about a week later and I was able to save everything on it. I still shake my head at that perfect timing. Of course, this meant I lost all the additional free space, haha. One step forward, two steps back...