3 ms·
How clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
by dylan604 4y ago
How clear cut is it when I'm storing a bunch of compressed video files? It's totally a waste at that point to even attempt to compress these files.
- treffer 4y agoThis is not how such storage systems work. If the whole storage system is for you and you only store compressed video files on it then perhaps. But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers. All the benefits hold, even for you with non compressible data, if there is an overall benefit. Hitting a bottleneck less often then this will reduce your tail latencies, too. It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.
- miohtama 4y agoVideo files have already entropy coding applied to them and thus any compression gains with reapplying a generic entropy coding like zstd are unlikely: https://en.wikipedia.org/wiki/Entropy_coding https://en.wikipedia.org/wiki/Entropy_coding
- sicp-enjoyer 4y agoThat's exactly what the poster is saying.
- goodpoint 4y agoNo, most compressed storage systems do not waste any CPU on data that is not compressible.
- treffer 4y agoHaving a big CPU (which you need to get many PCIe lanes) and then not using it is waste. It would be waste to _not_ use the CPU time. Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep) I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places. Do not waste the things you pay for!
- lazide 4y agoWhat I've personally implemented is trial compression with heuristics (you eagerly compress chunks, and if enough chunks don't compress, stop trying). It does require low level input/output control and per-chunk/block compression. That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.