3 ms·
We added zstd to largely supersede gzip use at work. It performs better than oldschool gzip in both compression and speed. That said, I'm not sure I'd call it
by loeg 9y ago
We added zstd to largely supersede gzip use at work. It performs better than oldschool gzip in both compression and speed.
That said, I'm not sure I'd call it a "real-time" compression algorithm. It's still an factor of 2x slower than lz4 compress and a factor of 4x slower decompress.
- michaelmior 9y agoIt's not clear exactly what "real-time" is supposed to mean, but I don't think it makes sense to define it in terms of other algorithms. If someone developed an algorithm faster than lz4, would lz4 cease to be real-time?
- joosters 9y agoI thought that 'real-time' is not about how fast a program is. Instead, it's about being able to guarantee that a program will always complete in under a fixed amount of time, under all circumstances. I can't see how that applies to Zstandard either.
- openasocket 9y agoHard real time certainly means that. I've seen "soft real-time" used to refer to programs where the standard deviation of the runtime given random input is low.
- michaelmior 9y agoYes, I that would be a correct definition although practically speaking the deadline that is guaranteed is important as well. Part of the problem is that the term "real-time" is overloaded.
- hinkley 9y agoIt’s a definition not used much anymore but this relates to on demand (streaming) versus ahead of time data processing. With only a small delay, can I compress and decompress at least one second of data per second, indefinitely. Think live TV versus film.
- chowells 9y agoThat's usually called an "online" algorithm, in my experience. It does seem to agree with the way it's used in this context.
- foota 9y agoOnline isn't quite the right distinction here imo, since that's really more about possibility than speed. An algorithm could be online (doesn't require all the data at once to run) but still not be fast enough to stream.
- hackcasual 9y agoTalking about compression you're generally talking about things that when you factor in network overhead have a neutral or better impact on end-to-end time.
- bgirard 9y agoIt's really subjective based on your computing resources client CPU + network speed and your server CPU if you can't compress ahead of time. Here's a great page that captures this: https://quixdb.github.io/squash-benchmark/#transfer-plus-processing https://quixdb.github.io/squash-benchmark/#transfer-plus-pro... TL;DR: Your optimal compression algorithm will vary based on these parameters. There's no universal answer.