4 ms·
> her work resulted in saving users an expected 1.5 petabytes (that's 1.5 million gigabytes) of data each day. I'm guessing this is not a measure of data at re
by jdcarter 10y ago
> her work resulted in saving users an expected 1.5 petabytes (that's 1.5 million gigabytes) of data each day.
I'm guessing this is not a measure of data at rest, but data transferred over the network. The couple samples listed on the page ranged from 2.5% improvement to 20.3% (vs. zLib) so I guess they're extrapolating that out to all app downloads and updates across the world. Nicely done.
More generally, we've seen some great advances in compression lately. I've been using Facebook's zStandard [1] for compression in a product I'm currently working on, and I've been extremely pleased with both its speed and compression ratio. The days of "just use zLib" are coming to a close.
[1]: https://github.com/facebook/zstd https://github.com/facebook/zstd
- rdtsc 10y agoAre you worried at all about their patents stance. I currently I think it says if you litigate with Facebook you lose the license. Otherwise I agree zstd is looking like a very nice improvement in an area where most people think nothing happens. I especially dictionary compression bit.
- jdcarter 10y agoThat's a good point, the downside of using new algorithms is playing by the rules of their patent licenses. In this situation I'm not concerned about litigation with Facebook, but I could foresee other products where that might be an issue. Aside: anywhere you store compressed data at rest, you need to store (in a header somewhere) the algorithm used to compress that data. If you need to change algorithms down the road--e.g. you do get into a lawsuit with Facebook--you'll need that header to know how to decompress old data vs. new.
- jontro 10y agoLooks like they're using the BSD license in this project, https://github.com/facebook/zstd/blob/dev/LICENSE https://github.com/facebook/zstd/blob/dev/LICENSE So no need to worry about the patent clause in this case right?