6 ms·

Perl is great, but today you can just as well (or easier) use ruby, python or even php for scripting one-liners and small tasks. However awk and sed are still f

by ivanhoe 13y ago

Perl is great, but today you can just as well (or easier) use ruby, python or even php for scripting one-liners and small tasks. However awk and sed are still fantastic option when you need just a quick processing on a command line. Pipe it in, do some transformations or replacements, pipe it out. Same goes for other unix power-tools like sort, unique, wc, head, tail, split, etc., they are all still very useful...

tux1968 13y ago
Well, I have close to 30 years Unix experience so you don't have to convince me of the power of a pipeline. But I do have to disagree with your affection for awk and sed even though in the past I have used them both extensively. The advantage of learning perl is that you can replace both utilities in your piped command line and only have to remember one syntax. And it integrates just as well as sed and awk do in any pipeline. But quite often, it will obviate the need for piping through other utilities. Perl is much closer to sed and awk than any of the other scripting languages you mention. As noted in other posts, you can even automatically convert awk scripts to perl. And it doesn't take much effort to convert sed scripts to perl where you can leverage more powerful pattern matching and procedural facilities to boot. If you already know sed and awk, by all means keep using them, they're fine. But if you're new to the Unix command line, you'll get the most bang for your buck by learning perl instead of awk and sed.
- ghshephard 13y ago
  As a sysadmin (not web/app or systems programmer) I spent about 7 years using Perl (wrote my company's HR <-> LDAP synchronization system in it) until Python came along (for me, around 2003), nowadays for one or two liners, it's sed/awk - the simplicity of awk '{print $1,$2}', awk '{tot+=$2; c++}END{print tot/c}' has just worked it's way into my fingers, I don't even have to think about sed 's/^.client="$.$",dev.*/\1/' foo.txt - I just look at the log file and the sed expression just appears at my fingertips. Anything more complex than that (unless it's a really obvious fit for awk) I switch over to Python. Its simple data model for things like HoA or AoH just imposes less cognitive overhead on me, and it's rich assembly of built in libraries make me productive from the get-go. This sort of thing would be incredibly difficult in awk - not sure if I would have to go to CPAN with perl - but I just assumed (and was correct) that python would have it built in. import json f=open("bus-stops.json") j=json.load(f) for a in j: print a['no'],",",a['lat'],",",a['lng'],",",a['name'] A simple "import json", "help(json)" at the Python command line, and 2 minutes later I was done. Also - I'm able to understand my code the next day - something I was never able to do with Perl, but for some reason I can with Python. I probably spend 90% of my time in sed/awk, and 10% of my time in Python. Haven't touched Perl in 10 years - not because it isn't an awesome language (it really is) - it's just that I have room in my head for one full blown language at a time, and Python replaced Perl for me.
  - tux1968 13y ago
    Yeah, i have no problem with python or any other language. Use what works for you. But perl does an acceptable job on every one of your examples.. perl -naE 'say $F[0], $F[1]' perl -nae '$tot+=$F[1]; $c++;END{print $tot/$c}' perl -ape 's/^.client="(.)",dev.*/\1/' foo.txt And finally: use JSON; use IO::All; use feature 'say'; $f = io('bus-stops.json')->all; $j = decode_json($f); $,=','; for (@$j) { say @$_{'no','lat','lng','name'}; } I still think newbies would get much more value out of learning perl basics than spending any time on sed/awk intricacies. But to each his own.
  - jnazario 13y ago
    awk tip: in your averaging example, the "c++" and reference to c in the END block can be replaced by the built-in var NR (for number of records). awk's builtins are useful.
berntb 13y ago
Good luck doing one liners with Python, as long as it use wspace indentation for e.g. loops and don't support {}:s as an option. :-( I'd argue with most others, Perl is the single most useful command line tool. Not the only one, of course. But afaik, you can't e.g. load a JSON lib in awk as part of a pipeline. (I deserialize dumped data structures multiple times a week in [ad hoc testing with] pipelined cmds.) Imho, if you know Ruby or PHP as the back of your hand, don't learn another scripting language for command line use. Learn some completely different language for some other use, instead.
- mtdewcmu 13y ago
  No, you can't work with JSON very easily in awk. Any kind of hierarchical format like JSON and XML will give you a headache in awk, and CSV can be difficult as well.
  - bsg75 13y ago
    While not as complete as a full CSV parsing lib, finding this made working with CSV in awk much easier: http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html http://www.gnu.org/software/gawk/manual/html_node/Splitting-... gawk -vFPAT='[^,]*|"[^"]*"' http://stackoverflow.com/questions/4205431/parse-a-csv-using-awk-and-ignoring-commas-inside-a-field/17287068#17287068 http://stackoverflow.com/questions/4205431/parse-a-csv-using...
    - berntb 13y ago
      That regexp fails with fields containing '"':s, but I guess you can grep for embedded double quotes ("") first. Are there multiple variants of coding '"' in CSV fields? I don't know -- but some people who do know are those who write the CSV libs I use! Edit: And as your link notes, it fails for embedded \n:s. Imnsho, awk needs csv (and json, etc) builtin, preferable as a plugin architecture. But then, why not just use the Perl superset?
      - mtdewcmu 13y ago

by ivanhoe 13y ago

tux1968 13y ago

Well, I have close to 30 years Unix experience so you don't have to convince me of the power of a pipeline. But I do have to disagree with your affection for awk and sed even though in the past I have used them both extensively. The advantage of learning perl is that you can replace both utilities in your piped command line and only have to remember one syntax. And it integrates just as well as sed and awk do in any pipeline. But quite often, it will obviate the need for piping through other utilities. Perl is much closer to sed and awk than any of the other scripting languages you mention. As noted in other posts, you can even automatically convert awk scripts to perl. And it doesn't take much effort to convert sed scripts to perl where you can leverage more powerful pattern matching and procedural facilities to boot. If you already know sed and awk, by all means keep using them, they're fine. But if you're new to the Unix command line, you'll get the most bang for your buck by learning perl instead of awk and sed.

ghshephard 13y ago
As a sysadmin (not web/app or systems programmer) I spent about 7 years using Perl (wrote my company's HR <-> LDAP synchronization system in it) until Python came along (for me, around 2003), nowadays for one or two liners, it's sed/awk - the simplicity of awk '{print $1,$2}', awk '{tot+=$2; c++}END{print tot/c}' has just worked it's way into my fingers, I don't even have to think about sed 's/^.client="$.$",dev.*/\1/' foo.txt - I just look at the log file and the sed expression just appears at my fingertips. Anything more complex than that (unless it's a really obvious fit for awk) I switch over to Python. Its simple data model for things like HoA or AoH just imposes less cognitive overhead on me, and it's rich assembly of built in libraries make me productive from the get-go. This sort of thing would be incredibly difficult in awk - not sure if I would have to go to CPAN with perl - but I just assumed (and was correct) that python would have it built in. import json f=open("bus-stops.json") j=json.load(f) for a in j: print a['no'],",",a['lat'],",",a['lng'],",",a['name'] A simple "import json", "help(json)" at the Python command line, and 2 minutes later I was done. Also - I'm able to understand my code the next day - something I was never able to do with Perl, but for some reason I can with Python. I probably spend 90% of my time in sed/awk, and 10% of my time in Python. Haven't touched Perl in 10 years - not because it isn't an awesome language (it really is) - it's just that I have room in my head for one full blown language at a time, and Python replaced Perl for me.
- tux1968 13y ago
  Yeah, i have no problem with python or any other language. Use what works for you. But perl does an acceptable job on every one of your examples.. perl -naE 'say $F[0], $F[1]' perl -nae '$tot+=$F[1]; $c++;END{print $tot/$c}' perl -ape 's/^.client="(.)",dev.*/\1/' foo.txt And finally: use JSON; use IO::All; use feature 'say'; $f = io('bus-stops.json')->all; $j = decode_json($f); $,=','; for (@$j) { say @$_{'no','lat','lng','name'}; } I still think newbies would get much more value out of learning perl basics than spending any time on sed/awk intricacies. But to each his own.
- jnazario 13y ago
  awk tip: in your averaging example, the "c++" and reference to c in the END block can be replaced by the built-in var NR (for number of records). awk's builtins are useful.

berntb 13y ago

Good luck doing one liners with Python, as long as it use wspace indentation for e.g. loops and don't support {}:s as an option. :-( I'd argue with most others, Perl is the single most useful command line tool. Not the only one, of course. But afaik, you can't e.g. load a JSON lib in awk as part of a pipeline. (I deserialize dumped data structures multiple times a week in [ad hoc testing with] pipelined cmds.) Imho, if you know Ruby or PHP as the back of your hand, don't learn another scripting language for command line use. Learn some completely different language for some other use, instead.

mtdewcmu 13y ago
No, you can't work with JSON very easily in awk. Any kind of hierarchical format like JSON and XML will give you a headache in awk, and CSV can be difficult as well.
- bsg75 13y ago
  While not as complete as a full CSV parsing lib, finding this made working with CSV in awk much easier: http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html http://www.gnu.org/software/gawk/manual/html_node/Splitting-... gawk -vFPAT='[^,]*|"[^"]*"' http://stackoverflow.com/questions/4205431/parse-a-csv-using-awk-and-ignoring-commas-inside-a-field/17287068#17287068 http://stackoverflow.com/questions/4205431/parse-a-csv-using...
  - berntb 13y ago
    That regexp fails with fields containing '"':s, but I guess you can grep for embedded double quotes ("") first. Are there multiple variants of coding '"' in CSV fields? I don't know -- but some people who do know are those who write the CSV libs I use! Edit: And as your link notes, it fails for embedded \n:s. Imnsho, awk needs csv (and json, etc) builtin, preferable as a plugin architecture. But then, why not just use the Perl superset?
    - mtdewcmu 13y ago