4 ms·
I think it is a clear cut, mostly because I do not think that compression compromises any of those features, all while making the user experience better. For a
by treffer 4y ago
I think it is a clear cut, mostly because I do not think that compression compromises any of those features, all while making the user experience better.
For any storage system like this you usually have a few bottlenecks. IO and Network are the obvious ones, followed by tiering (cache, fast io, slow io, ...) and at the very end CPU.
Now let's say network is your bottleneck. If you can send the data to the client in a compressed for then you get the compression ratio as additional bandwidth. And the user would get the data quicker! So compression to the network is a clear win.
But the common bottleneck is often IO, a high end SSDs with 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit link. That's without any redundancy over other overhead.
Again compression to the storage layer would decrease the total amount of IOs, thus making sure a customer gets data quicker.
Ok, let's say both are not the issue. The fastest compression algorithms compete with memcopy. So if you need just one copy of your data you might have been faster by compressing it.
Especially fast compression algorithms (zstd, lz4, snappy, lzo, ...) are worth the CPU cost with virtually no downsides. The problem is finding the right sweet spot that reduces the current bottleneck without creating a CPU bottleneck, but zstd offers the greatest flexibility there, too.
Oh for range requests.... Those large objects are likely split anyway, for easier error recovery (imagine 100MB into a 1GB transfer you notice that the file data was corrupted - not good). Once you work on blocks it's easy to do somewhat efficient range requests again.
- dylan604 4y agoHow clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
- treffer 4y agoThis is not how such storage systems work. If the whole storage system is for you and you only store compressed video files on it then perhaps. But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers. All the benefits hold, even for you with non compressible data, if there is an overall benefit. Hitting a bottleneck less often then this will reduce your tail latencies, too. It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.
- miohtama 4y agoVideo files have already entropy coding applied to them and thus any compression gains with reapplying a generic entropy coding like zstd are unlikely: https://en.wikipedia.org/wiki/Entropy_coding https://en.wikipedia.org/wiki/Entropy_coding
- sicp-enjoyer 4y agoThat's exactly what the poster is saying.
- goodpoint 4y agoNo, most compressed storage systems do not waste any CPU on data that is not compressible.
- treffer 4y agoHaving a big CPU (which you need to get many PCIe lanes) and then not using it is waste. It would be waste to _not_ use the CPU time. Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep) I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places. Do not waste the things you pay for!
- lazide 4y agoWhat I've personally implemented is trial compression with heuristics (you eagerly compress chunks, and if enough chunks don't compress, stop trying). It does require low level input/output control and per-chunk/block compression. That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.