3 ms·
Just yesterday I needed to extract, sort, and categorize the user agent strings for 6 months' traffic to a handful of sites (attempting to convince a company to
by thaumaturgy 8y ago
Just yesterday I needed to extract, sort, and categorize the user agent strings for 6 months' traffic to a handful of sites (attempting to convince a company to abandon TLS 1.0/1.1).
The first half of the job was exactly the process you described: start with one log file, craft a grep for it, craft a `grep -o` for the relevant part of each relevant line, add `sort | uniq -c | sort -r`, switch to zgrep for the archived rotated files, and so on.
The other half of the ass was done in a different language, using the output from the shell, because I needed to do a thousand or so lookups against a website and parse the results.
Composable shell tools is a very under-appreciated toolbox, IMO.
- svsucculents 8y agoThe one thing that I hate about 'half assed' comments when people refer to the unix shell is that the half ass really refers to users knowledge of how to actually construct efficient pipelines. How many times I've seen 'cat frick.csv | sed 's/foo/bar/g' | sort -n | uniq | cut -d "," -f 3' type nonsense masquerading as deeply effective and efficient extraction and transformation is mind blowing. Your example included.