terrelln

14 ms·

31.
by terrelln 3y ago
We also regularly run into hardware issues with Zstd. Often the decompressor is the first thing to interact with data coming over the network. Or like in this case the decompressor is generally very sensitive to bit-flips, with or without c
32.
by terrelln 3y ago
I've recently started playing Factorio for the first time, and was thinking about using SAT solvers for automated sub-factory blueprint design with beacons. E.g. for a given recipe how do I design a sub-factory that builds the recipe w
33.
by terrelln 3y ago
folly provides functions to resize std::string & std::vector without initialization [0]. [0] https://github.com/facebook/folly/blob/3c8829785e3ce86cb821c...
34.
by terrelln 3y ago
That said, this is an interesting article, and I love to see people experimenting with modern compression algorithms for package management! There are a lot of easy wins in this space.
35.
by terrelln 3y ago
In parts (1) and (2) comparing the default setting of Zstd (level 3) against the default setting of Brotli (level 11) is a bit misleading. It shows Brotli compressing ~30% better than Zstd, but Brotli's default level is >100x slower
36.
by terrelln 3y ago
Yeah, that is definitely a limitation with QAT. It isn't a great fit for larger data, as it can quickly lose compression ratio due to its smaller window. However, there is a lot of compression done on data that is <64KB. E.g. compre
37.
by terrelln 3y ago
Awesome post, TIL about that instruction. I just found myself wanting a `csinc` instruction when optimizing a function to merge sorted lists. Looking forward to your future posts!
38.
by terrelln 3y ago
This style of hash function is used by zstd, lz4, and many other LZ algorithms in their hash tables as a fast hash with good enough quality
39.
by terrelln 4y ago
I've seen a multi MB symbol coming from generated code
40.
Questions with the Zstandard open source team at Meta(youtube.com)
1 points by terrelln 4y ago|0 comments
41.
Questions with the Zstandard open source team at Meta(youtube.com)
3 points by terrelln 4y ago|0 comments
42.
by terrelln 4y ago
Awesome work! We were very happy to receive the patches to zstd to optimize ARM performance!
43.
by terrelln 4y ago
This is very interesting! I've tried something somewhat similar in the past. I was looking at implementing an extremely fast decompressor, with ratio similar to LZ4. I was able to get 2x the decompression speed of LZ4, but struggled wi
44.
by terrelln 4y ago
https://github.com/weissenberger/gpuhd The authors of this repo/paper use the self-synchronizing property of almost all Huffman codes to implement parallel Huffman decoding on the GPU at ~10 GB/s. In practice
45.
by terrelln 4y ago
It can still matter. You can have a common function that is inlined everywhere that takes a ton of CPU in aggregate, but each callsite is small. E.g Map::find().
46.
by terrelln 4y ago
Thanks for the pointer, will have to take a look!
47.
by terrelln 4y ago
Could you combine both techniques to run both the SIMD version on some chunks and the crc32 instruction on other chunks, in parallel? Of course this would only work if they execute on different ports.
48.
by terrelln 4y ago
Yeah thats correct. I'll just point out that they use a larger window size, so they will use more memory to decompress, but will still be fast.
49.
by terrelln 4y ago
Yes! You can build a decompressor only version of zstd that is only 95 KB with my version of gcc. > make -j libzstd ZSTD_LIB_MINIFY=1 ZSTD_LIB_COMPRESSION=0 ZSTD_LIB_DICTBUILDER=0 ZSTD_LEGACY_SUPPORT=0 ZSTD_LIB_DEPRECATED=0 &gt
50.
by terrelln 4y ago
We're still reserving the right to fiddle around with the meaning of our negative compression levels. We think that we may be able to offer more compression at the same speeds by completely changing our search strategies for very fast
51.
by terrelln 4y ago
Thanks for the feedback! I've opened an issue to track this [0] * Levels 1-19 are the "standard" compression levels. * Levels 20-22 are the "ultra" levels which require --ultra to use on the CLI. They allocate a lot
52.
by terrelln 4y ago
> Does zstd benefit today from ISAs like AVX-512, AVX-2, etc.? Zstd benefits mostly from BMI(2), it takes advantage of shlx, shrx, and bsf during entropy (de)coding. We do use SSE2 instructions in our lazy match finder to filter matches
53.
by terrelln 4y ago
Awesome post, it's great to see someone diving into the details of Zstd! Maintainer of Zstd here, AMA
54.
by terrelln 4y ago
Zstd has a 4-byte magic number, which is used to check if the data is zstd encoded. In addition to that, this example has a 2 byte frame header (including the decompressed size), a 3 byte block header, and a 4 byte checksum at the end (whic
55.
by terrelln 5y ago
Yeah, it is. The Linux kernel is currently using zstd-1.3.1, and I'm working on getting it updated to the latest zstd version.
56.
by terrelln 6y ago
I recently reworked how zstd builds its Huffman decoding tables (not decoding itself) to avoid unpredictable branches and speed the table building up by about ~2x [0]. This is insignificant for large decompressions, but if you're deco
57.
by terrelln 6y ago
Also, if you give zstd binary delta a try, we would be super interested in hearing the results you get! Especially around the tradeoff between compressed size, compression time, and decompression time. Opening an issue would be the best way
58.
by terrelln 6y ago
You're right that a totally fair comparison would use zstd. I didn't personally run these benchmarks, but I believe that the dominate time was computing the binary subtraction mappings, so I would expect similar results.
59.
by terrelln 6y ago
Zstd fully supports binary deltas. If you're computing a delta between `old` and `new` you would do this to round trip it: zstd --patch-from=old new -c | zstd -d --patch-from=old Using the library this can be achieved by using the old
60.
by terrelln 6y ago
One of the maintainers of zstd here, AMA

More ›