Ok, I'll bite again: marek:~$ cat logs-popcount-org.txt | perf stat -d awk '!a[$0] { a[$0]=1; print }' > /dev/null Performance counter stats for 'awk

Ok, I'll bite again: marek:~$ cat logs-popcount-org.txt | perf stat -d awk '!a[$0] { a[$0]=1; print }' > /dev/null Performance counter stats for 'awk !a[$0] { a[$0]=1; print }': 40,318.47 msec task-clock:u 0 context-switches:u 0 cpu-migrations:u 1,670,649 page-faults:u 112,979,634,215 cycles:u 93,441,976,758 instructions:u 18,990,099,679 branches:u 208,386,137 branch-misses:u 26,093,832,363 L1-dcache-loads:u 708,880,979 L1-dcache-load-misses:u 464,332,790 LLC-loads:u 245,913,835 LLC-load-misses:u 40.337768657 seconds time elapsed 36.851718000 seconds user 3.468126000 seconds sys Compare this to the optimized approach which has 57M LLC-load-misses, and 7M instructions.