4 ms·
LZ4 is amazing - it's basically the same idea as LZ77, only implemented carefully, with all the dozens parameters and tradeoffs made in a certain way. Couple th
by unsigner 5y ago
LZ4 is amazing - it's basically the same idea as LZ77, only implemented carefully, with all the dozens parameters and tradeoffs made in a certain way. Couple this with modern amounts of brute force thrown in an optimal parser (LZ4HC) and you get something quite extraordinary, on the "Pareto frontier" of compression efficiency. There's a tendency to focus on algorithms and scoff at implementation details, but LZ4 was possible as an algorithm 30 years ago, yet the implementation details crystallized to perfection only, say, five years ago.
- kevin_thibedeau 5y agoWe also have voluminous amounts of cache and multicore processors that can decompress in cache faster than reading uncompressed data. The tech did need to catch up.
- unsigner 5y agoTrue. If you're on your way down this rabbit hole, there's a bunch of old-machine-specific compression algorithms, developed by the emulator community, e.g. LZSA: https://github.com/emmanuel-marty/lzsa https://github.com/emmanuel-marty/lzsa
- cma 5y agoLZ4's choices may have also made less sense then due to memory latency, cache sizes, deep pipelining changing missed branch penalty, and other changes since the 70s.