Cyan4973

5 ms·

1.
by Cyan4973 7y ago
> Please be aware that this will significantly impact compression ratio. Is this a continuation from previous message ? Because multithreading mode of zstd doesn't impact its compression ratio. It's actually an interesting prop
2.
by Cyan4973 8y ago
I'm afraid I don't follow. Is there any code source that could be read ? The way to force the hash algorithm to actually wait for input is to use the previous hash to determine the start position of next hash's input. Jumping
3.
by Cyan4973 8y ago
Following your suggestion, I went ahead and modified the scrambling formula. The crux of the problem is that the secret is allowed to be absolutely anything (it's user provided). As a consequence, it's a bad idea to depend on it f
4.
by Cyan4973 8y ago
Actually, making the next hash dependent on previous hash through the `seed` doesn't work for benchmarking latency. Several hashes only use the seed at the very end of calculation. Among them, CityHash, FarmHash, MeowHash, probably a f
5.
by Cyan4973 8y ago
Thanks for suggestions @dragontamer. We are genuine when saying the algorithm is opened to suggestions, and can still change to improve its properties. Let's review yours : > my goal if I were to go through your code is to cut the s
6.
by Cyan4973 8y ago
> Why not optimize the function, and aim for only 256-bits of internal state (with 256-bits of entropy) ?? It's a lot more difficult to ensure that accumulators contain full-width entropy at all times. The trade-off in this algorith
7.
by Cyan4973 8y ago
Well, if you believe a better scrambling operation is possible, you are certainly welcomed to suggest one. Considering feedbacks on the algorithm is one of the objectives of the test phase. If the issue is about the default keys, it's
8.
by Cyan4973 8y ago
It's not exactly "losing information". We are not trying to regenerate original data, just make sure that all source bits can fairly influence the result. It's more a question of bit contribution. In the resulting 64-bit
9.
by Cyan4973 8y ago
According to the UMAC paper, the 32x32=>64 multiplication only contains 32-bit of entropy, even though it uses 64-bit space. That's understandable : most of the entropy will be in the middle of the register. That's enough for X
10.
by Cyan4973 8y ago
wyhash is a close relative to mumhashv2, and feature approximately the same performance strength and weaknesses.
11.
by Cyan4973 8y ago
On Windows, I'm using Chocolatey package manager : https://chocolatey.org/ zstd is among the available packages.
12.
by Cyan4973 8y ago
zstd availability is becoming more and more common nowadays. See for example this tracker : https://repology.org/metapackage/zstd/versions
13.
by Cyan4973 8y ago
There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper
14.
by Cyan4973 10y ago
There is now a pzstd implementation, if you wish to compare it to pigz
15.
by Cyan4973 10y ago
The decompressor can receive a parameter to refuse compressed frames requiring more than a selected amount of memory.
16.
by Cyan4973 10y ago
Zstd pass faster over incompressible data. Expect something > 1 GB/s
17.
by Cyan4973 10y ago
zstd goes at > 1 GB/s on uncompressible data. It has some fast heuristics for such cases too.
18.
by Cyan4973 11y ago
MetroHash source code is impressively close to xxHash one, to the point that it seems to be a tweaked version of it. Yet, the author never mentions a single word about it in its source.
19.
by Cyan4973 11y ago
In all tests I know, xxh64 should be about 2x faster than xxh32 on x86_64 systems. Surprisingly, it doesn't match with the readme table.