4 ms·

I have taken as rule to use awk only for trivial tasks and to switch to perl as soon as the syntax is slightly beyond my usual use cases. In perl, I would do:

I have taken as rule to use awk only for trivial tasks and to switch to perl as soon as the syntax is slightly beyond my usual use cases. In perl, I would do: perl -nle 'print unless exists $h{$_};$h{$_}++' < your_file

asicsp 7y ago
perl borrowed stuff from awk, so you could also do perl -ne 'print if !$seen{$_}++'
- rlonstein 7y ago
  Play a round of perl golf? perl -pne '$_=$#$_++?$_:""' I'm rusty at this but shaved off six chars, five if you count the 'p' added to switches.
  - asicsp 7y ago
    edit: this is failing if a line is repeated more than once, see @showdead's excellent explanation the shortest I've got so far is perl -lnE'say if!++$#$_' --- I don't understand what's happening with $#$_ but seems like something I should look into, thanks :) you could remove n as p is used and would be same no. of characters as perl -ne 'print if $#$_++' you could save one more by removing space between e switch and single quote
  - reacweb 7y ago
    I have never seen this $#$var trick and google is not the friend of perl operators. Do you have any explanation ? perl -ne 'print if!++$#$_' seems to work also
    - showdead 7y ago
      If you have array @foo in perl, $#foo is the index of the last element of the array, which is just the size of the array -1. So if @foo is undefined, $#foo is -1. Using a variable instead of 'foo' is a symbolic reference, so this is effectively using the symbol table as the associative array. This means that this solution also gets it wrong if your file contains a line that matches the name of a built-in variable in perl. That would be tough to debug! If your file contains This is the first line of the file then during execution of ++$#$_ the result is the same as if you had written ++$#{This is the first line of the file} So the variable @{This is the first line of the file} goes from undefined to an array of length 1, turning $#{This is the first line of the file} to 0. Incidentally, this is why the snippet fails to work for a line repeated more than once: for each occurrence of the expression, the value returned is in the sequence -1, 0, 1, 2, 3, ... so it is only false for the second occurrence. Using preincrement instead of postincrement means the values returned are 0, 1, 2, 3, ... which means that inverting the test makes it false for every occurrence after the first.
chasil 7y ago
There are many AWK implementations that are obscenely portable. A case in point is busybox AWK. Another might be http://unxutils.sourceforge.net http://unxutils.sourceforge.net A complex AWK script is very, very easy to move to a platform that lacks any AWK parsers. It can easily be done on Windows, without administrative rights, by the placement of a single .EXE - Perl can do many things, but that is not one of them (to the best of my knowledge).

asicsp 7y ago

perl borrowed stuff from awk, so you could also do perl -ne 'print if !$seen{$_}++'

rlonstein 7y ago
Play a round of perl golf? perl -pne '$_=$#$_++?$_:""' I'm rusty at this but shaved off six chars, five if you count the 'p' added to switches.
- asicsp 7y ago
  edit: this is failing if a line is repeated more than once, see @showdead's excellent explanation the shortest I've got so far is perl -lnE'say if!++$#$_' --- I don't understand what's happening with $#$_ but seems like something I should look into, thanks :) you could remove n as p is used and would be same no. of characters as perl -ne 'print if $#$_++' you could save one more by removing space between e switch and single quote
- reacweb 7y ago
  I have never seen this $#$var trick and google is not the friend of perl operators. Do you have any explanation ? perl -ne 'print if!++$#$_' seems to work also
  - showdead 7y ago
    If you have array @foo in perl, $#foo is the index of the last element of the array, which is just the size of the array -1. So if @foo is undefined, $#foo is -1. Using a variable instead of 'foo' is a symbolic reference, so this is effectively using the symbol table as the associative array. This means that this solution also gets it wrong if your file contains a line that matches the name of a built-in variable in perl. That would be tough to debug! If your file contains This is the first line of the file then during execution of ++$#$_ the result is the same as if you had written ++$#{This is the first line of the file} So the variable @{This is the first line of the file} goes from undefined to an array of length 1, turning $#{This is the first line of the file} to 0. Incidentally, this is why the snippet fails to work for a line repeated more than once: for each occurrence of the expression, the value returned is in the sequence -1, 0, 1, 2, 3, ... so it is only false for the second occurrence. Using preincrement instead of postincrement means the values returned are 0, 1, 2, 3, ... which means that inverting the test makes it false for every occurrence after the first.

chasil 7y ago

There are many AWK implementations that are obscenely portable. A case in point is busybox AWK. Another might be http://unxutils.sourceforge.net http://unxutils.sourceforge.net A complex AWK script is very, very easy to move to a platform that lacks any AWK parsers. It can easily be done on Windows, without administrative rights, by the placement of a single .EXE - Perl can do many things, but that is not one of them (to the best of my knowledge).