3 ms·
yeah, a generic, probably pointer-heavy hash table is definitely gonna be worse on memory. I'm surprised that it's that much worse on time though, I expected it
by Hello71 7y ago
yeah, a generic, probably pointer-heavy hash table is definitely gonna be worse on memory. I'm surprised that it's that much worse on time though, I expected it to be closer. I guess probably the cache misses are worse with such a large table though.
- majke 7y agoOk, I'll bite again: marek:~$ cat logs-popcount-org.txt | perf stat -d awk '!a[$0] { a[$0]=1; print }' > /dev/null Performance counter stats for 'awk !a[$0] { a[$0]=1; print }': 40,318.47 msec task-clock:u 0 context-switches:u 0 cpu-migrations:u 1,670,649 page-faults:u 112,979,634,215 cycles:u 93,441,976,758 instructions:u 18,990,099,679 branches:u 208,386,137 branch-misses:u 26,093,832,363 L1-dcache-loads:u 708,880,979 L1-dcache-load-misses:u 464,332,790 LLC-loads:u 245,913,835 LLC-load-misses:u 40.337768657 seconds time elapsed 36.851718000 seconds user 3.468126000 seconds sys Compare this to the optimized approach which has 57M LLC-load-misses, and 7M instructions.
- ncmncm 7y agoI would welcome seeing a comparison in your environment to using the simple 1/2 GB array of bits, with no hashing or storage of IP addresses. (Extra points for hugetlb mapping.)