3 ms·

Collation aside (which is absolutely a huge boost in speed that I neglected to think about), I assumed that the rest of the difference was coming from the fact

Collation aside (which is absolutely a huge boost in speed that I neglected to think about), I assumed that the rest of the difference was coming from the fact that the initial `cut` meant the rest of the pipeline had far less to deal with, whereas `awk` is processing every line. Benchmarking (and testing in `perf`) showed this to not be the case. I'd need to compile `awk` with debug symbols, I think, to know exactly where the slowdown is, but I'm going to assume it's mostly due to `sort` being extremely optimized for doing one thing, and doing it well. I did find one other interesting difference between BSD and GNU tools - BSD sort defaults to 90% for its buffer, GNU sort defaults to 1024 KiB. Combining all of these (and using GNU uniq - it was also faster), I was able to get down to 463 msec on the M1 Air: $ hyperfine "export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head" Benchmark 1: export LC_ALL=C; gcut -d' ' -f1 fake_log.txt | gsort -S5% | guniq -c | gsort -rn -S5% | head Time (mean ± σ): 463.4 ms ± 3.3 ms [User: 965.5 ms, System: 93.3 ms] Range (min … max): 459.9 ms … 469.8 ms 10 runs TIL, thank you.

xvector 3y ago
Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?
- sgarland 3y ago
  Parent poster used the 5% value, so I did as well. But yes, to answer your question. Shown here on a Debian 11 system that's fairly old: $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s] Range (min … max): 0.942 s … 1.937 s 10 runs $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s] Range (min … max): 3.591 s … 3.919 s 10 runs Setting the buffer value to ~half the file size (100 MB) resulted in a mean time of 2.291 seconds. Setting it to the size of the file resulted in a mean time of 1.549 seconds, which is close enough to the 5% to call it equal - not like this server isn't busy with other stuff, so it's hardly a good place for perfect benchmarking. Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.

xvector 3y ago

Could you elaborate on how you arrived at 5% for your buffer? Does specifying a buffer size really cause that much of a speed up?

sgarland 3y ago
Parent poster used the 5% value, so I did as well. But yes, to answer your question. Shown here on a Debian 11 system that's fairly old: $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort -S5% | uniq -c | sort -rn -S5% | head Time (mean ± σ): 1.504 s ± 0.318 s [User: 2.833 s, System: 0.474 s] Range (min … max): 0.942 s … 1.937 s 10 runs $ hyperfine "export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head" Benchmark 1: export LC_ALL=C; cut -d' ' -f1 fake_log.txt | sort | uniq -c | sort -rn | head Time (mean ± σ): 3.847 s ± 0.093 s [User: 4.165 s, System: 0.613 s] Range (min … max): 3.591 s … 3.919 s 10 runs Setting the buffer value to ~half the file size (100 MB) resulted in a mean time of 2.291 seconds. Setting it to the size of the file resulted in a mean time of 1.549 seconds, which is close enough to the 5% to call it equal - not like this server isn't busy with other stuff, so it's hardly a good place for perfect benchmarking. Syscalls aren't free, nor are disk reads, so if you have the RAM to support slurping the entire file at once, it's sometimes faster.