2 ms·
Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?
by xvector 3y ago
Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?
- sgarland 3y agoParent poster used the 5% value, so I did as well. But yes, to answer your question. Shown here on a Debian 11 system that's fairly old: $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s] Range (min … max): 0.942 s … 1.937 s 10 runs $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s] Range (min … max): 3.591 s … 3.919 s 10 runs Setting the buffer value to ~half the file size (100 MB) resulted in a mean time of 2.291 seconds. Setting it to the size of the file resulted in a mean time of 1.549 seconds, which is close enough to the 5% to call it equal - not like this server isn't busy with other stuff, so it's hardly a good place for perfect benchmarking. Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.