3 ms·

I have a project where I want two properties which are not inherently contradictory, but don't seem to be available together: 1. Huge compression window (like

by wyager 1y ago

I have a project where I want two properties which are not inherently contradictory, but don't seem to be available together: 1. Huge compression window (like 100+MB, so "chunking" won't work) 2. Random seeking into compressed payload Anyone know of any projects that can provide both of these at once?

wilted-iris 1y ago
That sounds like this with inline dictionaries added unless I’ve misunderstood you.
rorosen 1y ago
If seeking to frames is good enough for random seeking, this can be done with zeekstd already. The cli sets a custom window log (ZSTD_c_windowLog) on the compression context when creating binary patches[1], I regularly use it with a window size above 1G. [1] https://github.com/rorosen/zeekstd/blob/main/cli/src/compress.rs#L29 https://github.com/rorosen/zeekstd/blob/main/cli/src/compres...
- wyager 1y ago
  Nice, thanks.