klauspost

8 ms·

31.
by klauspost 4y ago
Here is an updated chart - and the data (2019): https://community.centminmod.com/threads/round-4-compression... Yes, though decompression speed (+memory to some extent) should also be considered. Also different input t
32.
by klauspost 4y ago
(article author here) Zstandard is good, no doubt. I didn't include compressor with entropy compression, except gzip which I added as a reference point for the ratios. After all I know it intimately after porting it to Go. In overall t
33.
by klauspost 4y ago
By "compressing" random data you are bypassing gzip, since it will just store your data as uncompressed blocks, making "decompression" a memcopy. With real data, deflate maxes out somewhere around there either way, but t
34.
by klauspost 4y ago
It is kind of a hack for decompression, where you look forward in the stream for a block signature, and try decompressing from there. In practice it works, but it isn't pretty ;)
35.
by klauspost 4y ago
Looks interesting, but my main objections to general adoption the same as bzip2, lzma and context modelling based codecs - decompression speed. Compressing logs for instance, decompression speed of 23MB/s per core, is simply too slow w
36.
by klauspost 4y ago
Yes, zstd has forced bitwise match coding, whereas lz4 is byte-aligned and with inline literals. So lz4 has some base advantages in terms of speed, which zstd is unlikely to match. But as you point out it is only relevant for very high spee
37.
by klauspost 5y ago
x86 Darwin had a bug where AVX512 K (opmask) registers would not be saved and restored on context switches when YMM registers had all 0 values. The kernel assumed that AVX512 wasn't used. They fixed it in Monterey 12.2, but still hasn&
38.
by klauspost 5y ago
No, S3, as MinIO, has a read-after-write consistency. So indexing would block on either writes or reads until it is done. We block when doing the zip indexing, but that is much more lightweight - and we limit to 100MB ZIP directory. That wa
39.
by klauspost 5y ago
Correct, but it pretty much forces a scan through the file. For AWS, you pay for scanning the file, even if only a few results are returned. Currently this is $0.002 per scanned GB, plus request and transfer fee. For MinIO you "pay&quo
40.
by klauspost 5y ago
> When it comes to stuff which compresses well but full of small objects, ZIP is pretty bad Correct, but it also allows you to independently access files, which is a win for this use-case. The goal isn't the compression itself, but
41.
by klauspost 5y ago
As mentioned in the blog post we do not expect to support TAR archives. There is no central directory in TAR, so indexing would be very expensive, and we would not be able to support tar.gz anyway. ZIP was chosen mainly because it fits with
42.
by klauspost 5y ago
Nice tools! When it is serverside, reading a 50MB CFD is a small task. And once it is read we can store the zipindex for even faster access. We made 'zipindex' to purposely be a sparse, compact, but still reasonably fast represent
43.
by klauspost 5y ago
We considered TAR, but indexing requires reading back and decompressing the entire archive. This may be feasible on small TAR files, and for single PutObject you could index while uploading. However for multipart objects, parts can arrive i
44.
by klauspost 5y ago
(dev here) Thanks! The initial idea was a customer request was to specify ranges as key=x->y;key2=z->w when uploading a file via a header and then specifying the key as a substitute for Range on get. We chewed a bit on that and found
45.
Zstandard Go Compression Package
4 points by klauspost 7y ago|0 comments
46.
Optimized Go gzip/zip packages, 30-50% faster(github.com)
2 points by klauspost 11y ago|0 comments
47.
Protect Your Passwords Against Dictionary Attacks(blog.klauspost.com)
1 points by klauspost 11y ago|0 comments
48.
by klauspost 11y ago
I like the language, so I just rather write a C/SSE/AVX library if I really need something to be very fast. I feel exactly the same, and once you get to know the quirks of the Go assembler it is actually pretty easy to use. That
49.
by klauspost 11y ago
Very nice work, and thanks for releasing under MIT. Cheers. The hard work goes to BB, their initial library, though :) Interesting that Backblaze decided to go with 17+3 configuration. Yeah. It seems they don't go for any RAID with
50.
by klauspost 11y ago
>one annoyance I have is that it seems there is no middle ground between alright performance using pure Go and great performance using Go with assembler. I don't think it is a Go-specific thing. It just happens that this is an alg