3 ms·
Moreover zstd quite unusual '--adapt' parameter enables it to "dynamically adapt compression level to perceived I/O conditions". Works for me (albeit the manpa
by natmaka 4y ago
Moreover zstd quite unusual '--adapt' parameter enables it to "dynamically adapt compression level to perceived I/O conditions". Works for me (albeit the manpage states that "it can remain stuck at low speed when combined with multiple worker threads").
- deleted 4y ago[deleted]
- metadat 4y agoToo bad the flag doesn't come with detection for this environmental condition and then coordinate accordingly across processes.
- thecleaner 4y agoIs there a paper on how it "perceives" the I/O conditions?
- flaviut 4y agoI'd guess by using backpressure. Modify the compression level to try and keep the output buffer at 60% full.
- natmaka 4y agoIt reacts to the input buffer state (in bad I/O conditions it starves). On Linux PSI ( /proc/pressure/io ) probably provides more accurate information (the code already uses /proc/cpuinfo ). Detail: in the fileio.c module there are lines such as: if (oldIPos == inBuff.pos) inputBlocked++; /* input buffer is full and can't take any more : input speed is faster than consumption rate / if ( (inputBlocked > inputPresented / 8) / input is waiting often, because input buffers is full : compression or output too slow */ This impacts a 'speedChange' variable. Its potential values (an enum) are 'noChange', 'slower', and 'faster'. They are processed rather simply: if (speedChange == slower) { ((...)) compressionLevel ++; if (speedChange == faster) { ((...)) compressionLevel --;