felixhandte

9 ms·

31.
by felixhandte 5y ago
Yes. For example, RocksDB uses zstd dictionaries: https://rocksdb.org/blog/2021/05/31/dictionary-compression.h...
32.
by felixhandte 5y ago
Two points worth noting: 1. Gzip is not a suitable compressor for this use case, because it's limited to a 32KB window. So the input can only be correlated with the last 32KB of the reference texts. 2. You can save a great deal in comp
33.
by felixhandte 5y ago
Glad to hear it! It's a pretty hefty single file, so it probably won't be qualifying for https://github.com/nothings/single_file_libs anytime soon... but hopefully people find it useful nonetheless.
34.
by felixhandte 6y ago
Do I also understand correctly that a certificate was issued recently based on proving ownership of a domain ten years ago? That also sounds like a serious issue.
35.
by felixhandte 6y ago
W.r.t. standard dictionaries, it's something we're interested in, but the fundamental reality of dictionaries is that their effectiveness is strongly tied to their specificity. Put another way, a universal dictionary is a self-con
36.
by felixhandte 6y ago
Yes. I work at Facebook on data compression. The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good er
37.
by felixhandte 6y ago
Browsers, no, they haven't shipped support at present. Facebook is using it for its HTTP traffic to its apps though. We've seen significant benefits. The compression ratio is pretty comparable to Brotli, but it decompresses very m
38.
by felixhandte 6y ago
Yeah, I've experimented. It is actually surprisingly close to native performance! A testament to the wasm engine, I guess. There's discussion on this topic here: https://github.com/facebook/zstd/issues&#x
39.
by felixhandte 6y ago
SpaceX did at one point describe using RCS to accelerate the paired vessels to facilitate fuel transfer [slide][video]. [slide] https://spaceflight101.com/spx/wp-content/uploads/sites/113/... [video
40.
by felixhandte 7y ago
Out of curiosity, why? Zstd performs as well as it does because of the enormous effort that has been invested into it. That necessarily brings complexity. We do actually have a simple “educational” decoder implementation [0], but I wouldn’t
41.
by felixhandte 7y ago
Yes. The 1.0.0 release froze the wire format (almost, see below). All Zstandard releases after that are all interoperable. It’s funny though that you should ask that question now. Just yesterday I did submit an erratum on the RFC [0], chang
42.
by felixhandte 7y ago
Yes! Three or more times faster.
43.
by felixhandte 7y ago
On my system, the gzip binary is 100KB and my zstd binary is 1.0MB. There are a number of reasons for this: - By default, the Zstd binary has a lot of additional functionality that gzip doesn't. For example: Zstd includes benchmarking
44.
by felixhandte 7y ago
Context: Zstandard is a compression algorithm invented by Yann Collet, with the goal to exceed zlib's compression performance along every dimension (those being compression ratio, compression speed, and decompression speed). Although t
45.
Zstandard v1.4.1(github.com)
60 points by felixhandte 7y ago|15 comments
46.
by felixhandte 7y ago
Just found the previous discussion from when they published the github repo: https://news.ycombinator.com/item?id=20412588 .
47.
Hermes: A JavaScript Engine Optimized for Mobile Apps(code.fb.com)
2 points by felixhandte 7y ago|1 comments
48.
by felixhandte 8y ago
This is a timely post, since IETF 104 is happening this week in Prague[1]. The QUIC working group will be meeting on Tuesday and Wednesday to make progress on standardization[2]. [1] https://datatracker.ietf.org/meeting&#x2F
49.
by felixhandte 8y ago
That is pretty dang fast!
50.
by felixhandte 8y ago
The Visual Display of Quantitative Information is one of my favorite books! 0 in this case is not really a relevant value (since that would mean transforming the input into something infinitely large). The functional identity value / o
51.
by felixhandte 8y ago
Thank you for helping build an ecosystem around Zstd!
52.
by felixhandte 8y ago
Given that it was Yann himself who wrote that sentence, I think that's a needlessly uncharitable interpretation. Maybe a better wording would have been "Two years ago, we released Zstandard v1.0, an open source ...". But I do
53.
by felixhandte 8y ago
Ha, that just came up: https://github.com/facebook/zstd/issues/1087
54.
by felixhandte 8y ago
Yep, that's in the vicinity of the solutions we're thinking about. There are a few proposals out there, and prior art like SDCH [1]. The hard part is that compression is already an attack vector for the web (e.g., CRIME [2], BREAC
55.
by felixhandte 8y ago
In short, we work with individual teams and projects to evaluate their priorities, and select or build the compression scheme that best addresses their needs. Some of those cases are described in the post, but we wanted to summarize the gen
56.
by felixhandte 8y ago
Hi. I made the graph. It was definitely not my intent to obscure or distort any information. Here's the raw data if you like [1]. Some context: the input text is silesia.tar, which is a standard mixed corpus of data. It was benchmarke
57.
by felixhandte 8y ago
It's something we're actively working on!
58.
by felixhandte 8y ago
At the moment, Brotli doesn't accept a user-provided dictionary. I know they're working to re-introduce that functionality, but it's not currently present.
59.
by felixhandte 8y ago
They compare pretty closely. The squash benchmark is a nice interactive comparison tool that usually agrees with my own benchmarks. See for example [1]. (Note that you should experiment with different input texts to get a sense for the vari
60.
Improving compression at scale with Zstandard(code.fb.com)
139 points by felixhandte 8y ago|87 comments

More ›