4 ms·
sort -u | wc -l but faster, with less memory, and slightly wrong
- vladf 5y agoOften, especially when analyzing production logs, you might be interested in questions like "how many users were active in the last hour?" or "how many active connections in the past hour for each of my servers during high load?". It's nice when you know this up-front and instrumented Prometheus accordingly, but often these questions are retrospective. If your log lines look like 010.100.111.132 - - [30/Jun/2021:15:30:11 +0000] "GET /rest/... HTTP/1.1" 200 0 "-" "User Agent" "-" 212.031.212.003 010.100.111.132 - - [30/Jun/2021:15:30:10 +0000] "GET /rest/... HTTP/1.1" 200 0 "-" "User Agent" "-" 212.031.212.004 010.100.111.132 - - [30/Jun/2021:15:30:09 +0000] "GET /rest/... HTTP/1.1" 200 0 "-" "User Agent" "-" 212.031.212.003 010.100.111.131 - - [30/Jun/2021:15:30:08 +0000] "GET /rest/... HTTP/1.1" 200 0 "-" "User Agent" "-" 212.031.212.006 Then you may be tempted to just use some `awk '{print $1 " " $15}' | sort -u | cut -d" " -f1 | uniq -c ` to back out unique IPs per server over the last hour for a rough estimate. By using sketches, we can scale this solution to cases where the number of log lines would make `sort` fail due to OOM or take forever. Here, you'd run `dsrs --key`.