4 ms·
>>> I use sed & awk all the time. Me too. For example, number of unique IP requests in a log file in milliseconds ;) $ awk '{ print $1 } ' caddy_log | sort |
by indescions_2017 9y ago
>>> I use sed & awk all the time.
Me too. For example, number of unique IP requests in a log file in milliseconds ;)
$ awk '{ print $1 } ' caddy_log | sort | uniq | wc -l
But rarely compose my own from scratch. It's mostly copy paste. And store in admin bin for future use.
- dietr1ch 9y agoThere's `sort -u` which should be able to avoid some work.
- ninkendo 9y agoDumb side note: I always end up using `awk '{print $1}'` instead of `cut -f1` because cut's field separation defaults are unwieldy in that it doesn't intelligently consider any whitespace to be a field separator, which is what I need 99% of the time.
- ptman 9y agosometimes (if there are not mixed spaces and tabs) tr -s ' ' < input | cut -f1 helps
- asicsp 9y agoNo need so many commands :) $ cat duplicates.txt abc 7 4 food toy **** abc 7 4 test toy 123 good toy **** $ awk '!seen[$2]++' duplicates.txt abc 7 4 food toy **** $ awk '!seen[$2]++{cnt++} END{print +cnt}' duplicates.txt 2
- dredmorbius 9y agoIf you just want to count unique IPs, you can get that in awk alone and avoid sort: awk '{ip[$1]=1}; END{ print length(ip) }' Use the hash tables directly, assuming you've got the memory. That should be slightly faster as it avoids the sort.
- asicsp 9y agonice one also, {ip[$1]} is enough...
- dredmorbius 9y agoThanks, I was trying to think of the fastest possible assignment.
- c4n4rd 9y agoNo need to spawn another process(uniq), you can use sort -u ;-)
- kbenson 9y agoFor me, the benefit of a trailing uniq instead of a sort -u is that I'm invariably building towards a "sort | uniq -c | sort -n" anyways.