5 ms·

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

partycoder 8y ago
Can also use comm <flags> <(command 1) <(command 2) To use the output of a command as input. This also works with diff and other commands.
- TheDong 8y ago
  More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again) The '<( ... )' is just giving the path to that command's stdout as a file descriptor. $ ls <(echo hi) /proc/self/fd/11 $ vi <(echo hi) # opens vi with 'hi' as the contents
  - therein 8y ago
    Thank you. I was just thinking of looking into how it was implemented.
    - lloeki 8y ago
      It is called "process substitution", and can work the other way around: http://www.tldp.org/LDP/abs/html/process-sub.html http://www.tldp.org/LDP/abs/html/process-sub.html With fish: https://fishshell.com/docs/current/commands.html#psub https://fishshell.com/docs/current/commands.html#psub But there it only works one way: https://github.com/fish-shell/fish-shell/issues/1786 https://github.com/fish-shell/fish-shell/issues/1786
- scintill76 8y ago
  Note that this is a bash feature; other shells may have different syntax.
  - hawski 8y ago
    It's the same in ksh at least, so one is covered in the BSDs. It could be also implemented as a separate tool, but not as convenient, because of additional quoting - I did once.
- hawski 8y ago
  I use it in my frugal SSG project [0] to remove HTML files for every removed source file. It's great to use it this way as it doesn't use temp files. [0] https://gist.github.com/hadrianw/060944011acfcadd889d937b960a5a1f https://gist.github.com/hadrianw/060944011acfcadd889d937b960...
deleted 8y ago
[deleted]
buddydvd 8y ago
I tried out your command and got different line count compared to the output of the commands mentioned in the post. The man page for comm says it assumes inputs to be pre-sorted. To do that, you need to sort the input files before passing them to comm. # show only items in both a and b comm -1 -2 <(sort -u a_list) <(sort -u b_list) # show only items unique to a comm -2 -3 <(sort -u a_list) <(sort -u b_list) # show only items unique to b comm -1 -3 <(sort -u a_list) <(sort -u b_list)
- sciurus 8y ago
  Yes, pre-sorting is necessary for comm to work. I didn't mention that because it's necessary for the commands in the post to work too.
alpb 8y ago
Indeed comm will work much faster than uniq in this case, because it reads each file only once, and does a single pass on each file. What the blog post does (doing cat multiple times, and then counting the occurrences) is going to be much slower, although mathematically correct.
- fiddlerwoaroof 8y ago
  Doesn't uniq only read the input once? I've always assumed that the reason uniq assumes its input is presorted is that, then, it doesn't need to buffer all of it: instead, it just eliminates successive duplicate lines.
  - alpb 8y ago
    Yeah but in this case the author is doing "cat 1.txt 1.txt 2.txt".
dceddia 8y ago
I love how, even after using Unix-y systems for probably 10-odd years, I still occasionally stumble upon something like this that refers to a command that I've never even heard of, and when I check (on my Mac, no less), it's right there sitting in /usr/bin. So many useful little utilities. I think the same thing happens to a lesser degree with vim features.
- bewuethr 8y ago
  A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently. Other boring, but instructive reads (heavily GNU biased): diffutils [1], findutils [2], the Bash manual [3]. [0]: https://www.gnu.org/software/coreutils/manual https://www.gnu.org/software/coreutils/manual [1]: https://www.gnu.org/software/diffutils/manual https://www.gnu.org/software/diffutils/manual [2]: https://www.gnu.org/software/findutils/manual/find.html https://www.gnu.org/software/findutils/manual/find.html [3]: https://www.gnu.org/software/bash/manual https://www.gnu.org/software/bash/manual
  - kqr 8y ago
    The chapter on shell utilities in the POSIX standard is another goldmine!
  - sciurus 8y ago
    I'd also recommend Unix Power Tools. There's definitely some outdated content, but I learned a lot about shell usage and text manipulation from it. http://shop.oreilly.com/product/9780596003302.do http://shop.oreilly.com/product/9780596003302.do
    - Anthony-G 8y ago
      Seconded. The dead tree copy I purchased in the mid-2000s is the best technical book in my collection. I’ve read most of it and every so often I still dip in to learn something new or – more often than not – to re-learn something I’ve forgotten.
raverbashing 8y ago
comm is probably the most useful command I never remember how it is called (and apropos doesn't help).
- paxunix 8y ago
  The only options you usually need for comm are -1, -2, and -3. It takes two arguments, call them 1 and 2. Its job is to include lines from each and both files in output (by default, you get 3 columns of output), so we want to filter by telling it to exclude what we don't want. My mnemonic is "-" means "not", so -1 means "not only in file 1", -2 means "not only in file 2", and -3 means "not in both". So now, when I want comm output that shows the lines unique to my 2nd file, it's: comm "not only in 1" and "not in both" so: comm -1 -3 file1 file2
  - raverbashing 8y ago
    I know how to use 'man', what I mean is, I never remember the command name (comm) because I always remember it's a command to compare so I think of cmp, and not comm.

partycoder 8y ago

Can also use comm <flags> <(command 1) <(command 2) To use the output of a command as input. This also works with diff and other commands.

TheDong 8y ago
More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again) The '<( ... )' is just giving the path to that command's stdout as a file descriptor. $ ls <(echo hi) /proc/self/fd/11 $ vi <(echo hi) # opens vi with 'hi' as the contents
- therein 8y ago
  Thank you. I was just thinking of looking into how it was implemented.
  - lloeki 8y ago
    It is called "process substitution", and can work the other way around: http://www.tldp.org/LDP/abs/html/process-sub.html http://www.tldp.org/LDP/abs/html/process-sub.html With fish: https://fishshell.com/docs/current/commands.html#psub https://fishshell.com/docs/current/commands.html#psub But there it only works one way: https://github.com/fish-shell/fish-shell/issues/1786 https://github.com/fish-shell/fish-shell/issues/1786
scintill76 8y ago
Note that this is a bash feature; other shells may have different syntax.
- hawski 8y ago
  It's the same in ksh at least, so one is covered in the BSDs. It could be also implemented as a separate tool, but not as convenient, because of additional quoting - I did once.
hawski 8y ago
I use it in my frugal SSG project [0] to remove HTML files for every removed source file. It's great to use it this way as it doesn't use temp files. [0] https://gist.github.com/hadrianw/060944011acfcadd889d937b960a5a1f https://gist.github.com/hadrianw/060944011acfcadd889d937b960...

deleted 8y ago

[deleted]

buddydvd 8y ago

I tried out your command and got different line count compared to the output of the commands mentioned in the post. The man page for comm says it assumes inputs to be pre-sorted. To do that, you need to sort the input files before passing them to comm. # show only items in both a and b comm -1 -2 <(sort -u a_list) <(sort -u b_list) # show only items unique to a comm -2 -3 <(sort -u a_list) <(sort -u b_list) # show only items unique to b comm -1 -3 <(sort -u a_list) <(sort -u b_list)

sciurus 8y ago
Yes, pre-sorting is necessary for comm to work. I didn't mention that because it's necessary for the commands in the post to work too.

alpb 8y ago

Indeed comm will work much faster than uniq in this case, because it reads each file only once, and does a single pass on each file. What the blog post does (doing cat multiple times, and then counting the occurrences) is going to be much slower, although mathematically correct.

fiddlerwoaroof 8y ago
Doesn't uniq only read the input once? I've always assumed that the reason uniq assumes its input is presorted is that, then, it doesn't need to buffer all of it: instead, it just eliminates successive duplicate lines.
- alpb 8y ago
  Yeah but in this case the author is doing "cat 1.txt 1.txt 2.txt".

dceddia 8y ago

I love how, even after using Unix-y systems for probably 10-odd years, I still occasionally stumble upon something like this that refers to a command that I've never even heard of, and when I check (on my Mac, no less), it's right there sitting in /usr/bin. So many useful little utilities. I think the same thing happens to a lesser degree with vim features.

bewuethr 8y ago
A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently. Other boring, but instructive reads (heavily GNU biased): diffutils [1], findutils [2], the Bash manual [3]. [0]: https://www.gnu.org/software/coreutils/manual https://www.gnu.org/software/coreutils/manual [1]: https://www.gnu.org/software/diffutils/manual https://www.gnu.org/software/diffutils/manual [2]: https://www.gnu.org/software/findutils/manual/find.html https://www.gnu.org/software/findutils/manual/find.html [3]: https://www.gnu.org/software/bash/manual https://www.gnu.org/software/bash/manual
- kqr 8y ago
  The chapter on shell utilities in the POSIX standard is another goldmine!
- sciurus 8y ago
  I'd also recommend Unix Power Tools. There's definitely some outdated content, but I learned a lot about shell usage and text manipulation from it. http://shop.oreilly.com/product/9780596003302.do http://shop.oreilly.com/product/9780596003302.do
  - Anthony-G 8y ago
    Seconded. The dead tree copy I purchased in the mid-2000s is the best technical book in my collection. I’ve read most of it and every so often I still dip in to learn something new or – more often than not – to re-learn something I’ve forgotten.

raverbashing 8y ago

comm is probably the most useful command I never remember how it is called (and apropos doesn't help).

paxunix 8y ago
The only options you usually need for comm are -1, -2, and -3. It takes two arguments, call them 1 and 2. Its job is to include lines from each and both files in output (by default, you get 3 columns of output), so we want to filter by telling it to exclude what we don't want. My mnemonic is "-" means "not", so -1 means "not only in file 1", -2 means "not only in file 2", and -3 means "not in both". So now, when I want comm output that shows the lines unique to my 2nd file, it's: comm "not only in 1" and "not in both" so: comm -1 -3 file1 file2
- raverbashing 8y ago
  I know how to use 'man', what I mean is, I never remember the command name (comm) because I always remember it's a command to compare so I think of cmp, and not comm.