3 ms·
There is a "!" missing from the title[0], should be awk '!a[$0]++' "removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which
by yesenadam 6y ago
There is a "!" missing from the title[0], should be
awk '!a[$0]++'
"removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which either removes only adjacent duplicates or has to break order to remove duplicates."
Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be printed/saved:
gawk -i inplace '!a[$0]++' file
(The original version seems to only work with files up to about 2GB.)
p.s. The current misspelled title using "Awk", raises another question–I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
[0] I sympathize—I once tried to submit a !!Con talk.
[1] https://unix.stackexchange.com/a/320128 https://unix.stackexchange.com/a/320128
- bewuethr 6y agoAs for "awk", "Awk", and "AWK" all working on macOS: APFS seems to default to being case-insensitive.
- whymarrh 6y ago> I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before! I presume this is a side-effect of the macOS default filesystem being case-insensitive. I'm running macOS with a case-sensitive fs and that does not work: $ awk usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...] $ Awk -bash: Awk: command not found This default is a topic of great debate.
- yesenadam 6y agoThis explains how "-i inplace" works: https://www.gnu.org/software/gawk/manual/html_node/Extension-Sample-Inplace.html https://www.gnu.org/software/gawk/manual/html_node/Extension...