terrelln

11 ms·

61.
by terrelln 6y ago
Lets say you have 8 independent counters and 4 CPUs. Each gets 4 slots in 4-cacheline = 256 byte storage. Counter0 gets slots 0, 64, 128, and 192. Counter1 gets slots 8, 72, 136, and 200. And so on. Then you map CPU0 to slot 0, CPU1 to slot
62.
by terrelln 6y ago
That’s true. But, I can’t tell all applications that use the counter library to reduce the number of threads they use.
63.
by terrelln 6y ago
Yeah that’s right. All the counters use the same indexing scheme, so all accesses to the same cache line should be on the same core.
64.
by terrelln 6y ago
I'm hoping to eventually migrate to restartable sequences.
65.
by terrelln 6y ago
I've implemented counters exactly like this. I actually migrated away from the tls_counter to something like the cas_multi_counter. The problem with the tls_counter is memory usage that scales with the number of threads. In this exampl
66.
Build a do-it-yourself air purifier for about $25(youtube.com)
2 points by terrelln 6y ago|0 comments
67.
by terrelln 6y ago
Convince means the verifier can prove to itself what the provers are telling it in polynomial time, without trusting the provers.
68.
by terrelln 7y ago
Zstd by default will write sparse files when it detects runs of zeros during decompression, but isn't aware of it when reading a file.
69.
by terrelln 7y ago
Zstd has a 32-bit checksum over the uncompressed data, which is enabled by default on the CLI.
70.
by terrelln 7y ago
This is an important trick for LZ4's speed. But, there is a bit more to it than this. Some important tricks are: * LZ4 can represent a literal length of 14, and a match length of 18 in a single byte [0]. Most LZ4 sequences will have a
71.
by terrelln 7y ago
Slightly off topic, but one great way to diff is to compress the new file using the old file as a dictionary. Both zlib and zstd support this (but zlib is limited to a 32KB file). You can tune how much CPU you want to spend getting a better
72.
Arch Linux’s pacman-5.2 to use zstd compression(archlinux.org)
1 points by terrelln 7y ago|0 comments
73.
by terrelln 7y ago
Zstd also has a multithreaded mode. On the CLI `zstd -T0` tells zstd to use all available cores.
74.
by terrelln 7y ago
Zstd's dictionaries contain two things: 1. Predefined statistics based on the training data for literals (bytes we couldn't find matches for), literal lengths, match lengths, and offset codes. These allow us to use tuned statistic
75.
by terrelln 7y ago
In this competition, and in similar competitions, the size of the binary used to decompress is taken into account. If you wanted to use a dictionary, you would need to pay for it in binary size. In this competition, the file must be self-de
76.
by terrelln 7y ago
Brotli is great at compressing static web content, zstd without a dictionary is unlikely to outperform it. For static content you'd probably rather save 5% of space over some decompression costs, since Brotli decompression is fast enou
77.
by terrelln 7y ago
lz4 compresses and decompresses faster than snappy, and compresses similarly. You can see some comparisons on the GitHub's readme https://github.com/lz4/lz4 .
78.
by terrelln 7y ago
There aren't any technical limitations to adding multithreaded decompression zstd. We just need a compelling enough use case to justify the work it would take to add it. pzstd is now obsoleted by zstd -T0, but it offers multithreaded d
79.
by terrelln 7y ago
A comparable zstd call that uses a 64 MB window size and all cores is: zstd --long=26 -T0 From there you can tune the compression level, or increase the window size up to 2 GB (--long=31). zstd won't beat the compression of xz
80.
by terrelln 7y ago
Starting with zstd-1.3.8 we support the `ZSTD_CLEVEL` environment variable. We've started with a small scope for the variable, because we don't want users to unexpectedly remove the source fie, for instance. If you want to pass ex
81.
by terrelln 7y ago
tar-1.3.1 added support for zstd with the option `--zstd` and `-a`, the auto-decompression flag, also supports zstd. Older tar versions also have the `-I` flag which you can use to (de)compress with zstd. We're working to improve zstd
82.
by terrelln 7y ago
The next GRUB release (grub-2.04) includes my patch to add support for zstd compressed BtrFS filesystems, which should solve one of the major pain points of Zstd BtrFS compression.
83.
by terrelln 7y ago
Disclaimer: I'm a maintainer of zstd, so I'm biased. Brotli dominates HTTP compression. Zstd just got its RFC approved a few months ago, but Brotli has been present in browsers for years. However, zstd is more widely adopted every
84.
Zstandard v1.4.0(github.com)
158 points by terrelln 7y ago|67 comments
85.
by terrelln 8y ago
The images are high res, but you have to download them to view the larger image.
86.
by terrelln 8y ago
I believe that this is just the initial release for testing, and the author plans to clean up the header file to separate out the interface from the implementation, and add documentation.
87.
by terrelln 8y ago
I believe you are looking at XXH64 and XXH32, which is the old version. The new version XXH3 is located in https://github.com/Cyan4973/xxHash/blob/dev/xxh3.h which exposes the public prototypes `XXH3_64b
88.
by terrelln 8y ago
I've tested XXH3 using xxhash's built-in benchmark tool with clang-7.0.1 and gcc-8.2.1 on an Intel i9-9900K. The processor was otherwise idle, and was running at 5 GHz. The command I tested is `xxhsum -b5i10`. SSE2: CFLAGS=-
89.
by terrelln 8y ago
That site claims 0.99 BPC, which I guess means the compressed size is 10^8 * 0.99 / 8 ~= 12,375,000. Does that include the size of the model, or is this just the cost to encode the errors?
90.
by terrelln 8y ago
I definitely agree that parallelism isn’t held back by fundamentally serial work. Even with LZ decompression, most algorithms entropy encode the LZ sequences, and these sequences can be decoded in parallel with the LZ decoder.

More ›