7 ms·
I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can
by jph 8y ago
I like shell set operations scripts, because they are quick and easy.
I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating.
Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome.
https://github.com/sixarm/setop https://github.com/sixarm/setop
#!/bin/sh
set -eu
op="$1"; shift
case $op in
∪|u|union|or|∨|add|addition|'+'|'|')
awk '!seen[$0] {print} {seen[$0]=1}' "$@"
;;
∩|i|intersection|and|∧|'&')
awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==argind) { print key } } }' "$@"
;;
⊖|d|diff|difference|xor|⊻)
awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==1) { print key } } }' "$@"
;;
ex|except|exclude|subtract|subtraction|'-')
awk 'NR==FNR{seen[$0]=1;next} seen[$0]=0; END { for (key in seen) { if (seen[key]) { print key } } }' "$@"
;;
extra)
awk 'BEGIN{argindmax=ARGC-1} FNR==1{argind+=1} argind<argindmax {seen[$0]; next}!($0 in seen)' "$@"
;;
disjoint|n|not|none)
awk 'seen[$0]==1 {x=1;exit} {seen[$0]=1} END { print x ? "FALSE" : "TRUE"}; exit !!x}' "$@"
;;
*)
esac
- FractalLP 8y agoHow long have you been using Awk?
- jph 8y agoFor a few years for two clients (financial and governmental) that have a range of older controlled systems. We benefit from simple analytics that are POSIX, not python, perl, R, etc. http://www.numcommand.com/ http://www.numcommand.com/
- unixhero 8y ago'Hat tip
- FractalLP 8y agoI've never heard of numcommand before, but it looks super awesome and exactly what I need. I use Linux servers, but am forced to use Windows as my local. So first question, do you have experience with awk/gawk on Windows? Second question: What do you prefer about Awk over Python/Perl? I guess if all your analytics are fairly short it is just easier in Awk? Thanks!
- jph 8y agoThank you! Yes in my experience of awk/gawk on Windows, things work the same, other than file separators, line endings, and similar platform-specific issues. For your second question, awk simply works. It's reliable, fast, and everywhere. I also like python and perl, and choose these for any modern system.
- FractalLP 8y agoThank you!
- fiddlerwoaroof 8y agodoesn't this read the entire input into memory? `uniq` and `comm` don't (need to) do this, so they can work on inputs bigger than available memory.
- jph 8y agoYes in the `setop` current implementation, because this enables the script to be fast, short, and work with inputs without needing to call `sort`. For comparison, a typical POSIX `uniq` implementation reads the input and solely compares two adjacent lines; this requires the input to be presorted. An interesting upgrade could be to add a `setop` option flag that tells the script the inputs are already sorted and/or deduped. This can achieve the memory savings you're describing.
- int0x80 8y agoThats nice, so is the article's examples. But what I always need to lookup is how to make the operations work on especific columns, rather than on whole lines. I mean without resorting to awk.
- unixhero 8y agoA few comments. First, wow, I love how this looks like dark magic. Those mathematical notations, are you using them because it makes it easier to see how it corresponds to actual Set Theory/theorems? If so, could you just as well have used an alphanummeric identifier like "left" "union" "right" or - would the code break without this notation? I'm on deep waters here, I don't know this. But set theory seems to pop up a lot in my line of work, essentially doing joins in datasets using Tableau - so my interest in the nitty gritty of this field is increasing.
- jph 8y agoThanks! To answer your questions... > because it makes it easier to see how it corresponds to actual Set Theory/theorems? Yes. These are the Unicode symbols for set theory. > could you just as well have used an alphanummeric identifier like "left" "union" "right" or Yes. You can use any of the words in the case switch statements, such as `setop union file1 file2`. You can also edit the script to add your own words if you like. You can see simpler versions of these scripts in our GitHub repos. For example the `union` command is https://github.com/sixarm/union https://github.com/sixarm/union > set theory seems to pop up a lot in my line of work More and more in mine too. Thank you for your comments!
- amelius 8y agoCould you add a "--help" option? Or provide usage instructions in a comment?