4 ms·

Having used bash, awk, sed and a hodgepodge of utilities for twenty years, it was a revelation to finally spend some time learning perl. It's not a panacea but

by tux1968 13y ago

Having used bash, awk, sed and a hodgepodge of utilities for twenty years, it was a revelation to finally spend some time learning perl. It's not a panacea but it does offer a much larger return for time invested in learning the basics. It is today as ubiquitous as sed & awk, and provides much more power and sophistication for no extra complexity. $0.02

cstross 13y ago
Well, considering the original remit for perl (back in the pre-Perl 5, pre-OOP, pre-CPAN days) was to glue sed and awk together with a proper programming language and full access to the POSIX library, that shouldn't be a surprise. Aren't a2p and s2p still part of the standard Perl distribution? (Haven't looked: stopped driving Perl for a living more than ten years ago.) If there's one argument against using perl in place of these other tools it's simply the cognitive overhead of learning the extra stuff perl brings to the table on top of bash, sed and awk. On the other hand, picking up non-OO old-school perl if you're already a proficient shell scripter should be a day's work ... the important thing is knowing when to switch tools.
ivanhoe 13y ago
Perl is great, but today you can just as well (or easier) use ruby, python or even php for scripting one-liners and small tasks. However awk and sed are still fantastic option when you need just a quick processing on a command line. Pipe it in, do some transformations or replacements, pipe it out. Same goes for other unix power-tools like sort, unique, wc, head, tail, split, etc., they are all still very useful...
- tux1968 13y ago
  Well, I have close to 30 years Unix experience so you don't have to convince me of the power of a pipeline. But I do have to disagree with your affection for awk and sed even though in the past I have used them both extensively. The advantage of learning perl is that you can replace both utilities in your piped command line and only have to remember one syntax. And it integrates just as well as sed and awk do in any pipeline. But quite often, it will obviate the need for piping through other utilities. Perl is much closer to sed and awk than any of the other scripting languages you mention. As noted in other posts, you can even automatically convert awk scripts to perl. And it doesn't take much effort to convert sed scripts to perl where you can leverage more powerful pattern matching and procedural facilities to boot. If you already know sed and awk, by all means keep using them, they're fine. But if you're new to the Unix command line, you'll get the most bang for your buck by learning perl instead of awk and sed.
  - ghshephard 13y ago
    As a sysadmin (not web/app or systems programmer) I spent about 7 years using Perl (wrote my company's HR <-> LDAP synchronization system in it) until Python came along (for me, around 2003), nowadays for one or two liners, it's sed/awk - the simplicity of awk '{print $1,$2}', awk '{tot+=$2; c++}END{print tot/c}' has just worked it's way into my fingers, I don't even have to think about sed 's/^.client="$.$",dev.*/\1/' foo.txt - I just look at the log file and the sed expression just appears at my fingertips. Anything more complex than that (unless it's a really obvious fit for awk) I switch over to Python. Its simple data model for things like HoA or AoH just imposes less cognitive overhead on me, and it's rich assembly of built in libraries make me productive from the get-go. This sort of thing would be incredibly difficult in awk - not sure if I would have to go to CPAN with perl - but I just assumed (and was correct) that python would have it built in. import json f=open("bus-stops.json") j=json.load(f) for a in j: print a['no'],",",a['lat'],",",a['lng'],",",a['name'] A simple "import json", "help(json)" at the Python command line, and 2 minutes later I was done. Also - I'm able to understand my code the next day - something I was never able to do with Perl, but for some reason I can with Python. I probably spend 90% of my time in sed/awk, and 10% of my time in Python. Haven't touched Perl in 10 years - not because it isn't an awesome language (it really is) - it's just that I have room in my head for one full blown language at a time, and Python replaced Perl for me.
    - tux1968 13y ago
      Yeah, i have no problem with python or any other language. Use what works for you. But perl does an acceptable job on every one of your examples.. perl -naE 'say $F[0], $F[1]' perl -nae '$tot+=$F[1]; $c++;END{print $tot/$c}' perl -ape 's/^.client="(.)",dev.*/\1/' foo.txt And finally: use JSON; use IO::All; use feature 'say'; $f = io('bus-stops.json')->all; $j = decode_json($f); $,=','; for (@$j) { say @$_{'no','lat','lng','name'}; } I still think newbies would get much more value out of learning perl basics than spending any time on sed/awk intricacies. But to each his own.
    - jnazario 13y ago
      awk tip: in your averaging example, the "c++" and reference to c in the END block can be replaced by the built-in var NR (for number of records). awk's builtins are useful.
- berntb 13y ago
  Good luck doing one liners with Python, as long as it use wspace indentation for e.g. loops and don't support {}:s as an option. :-( I'd argue with most others, Perl is the single most useful command line tool. Not the only one, of course. But afaik, you can't e.g. load a JSON lib in awk as part of a pipeline. (I deserialize dumped data structures multiple times a week in [ad hoc testing with] pipelined cmds.) Imho, if you know Ruby or PHP as the back of your hand, don't learn another scripting language for command line use. Learn some completely different language for some other use, instead.
  - mtdewcmu 13y ago
    No, you can't work with JSON very easily in awk. Any kind of hierarchical format like JSON and XML will give you a headache in awk, and CSV can be difficult as well.
    - bsg75 13y ago
      While not as complete as a full CSV parsing lib, finding this made working with CSV in awk much easier: http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html http://www.gnu.org/software/gawk/manual/html_node/Splitting-... gawk -vFPAT='[^,]*|"[^"]*"' http://stackoverflow.com/questions/4205431/parse-a-csv-using-awk-and-ignoring-commas-inside-a-field/17287068#17287068 http://stackoverflow.com/questions/4205431/parse-a-csv-using...
      - berntb 13y ago
        That regexp fails with fields containing '"':s, but I guess you can grep for embedded double quotes ("") first. Are there multiple variants of coding '"' in CSV fields? I don't know -- but some people who do know are those who write the CSV libs I use! Edit: And as your link notes, it fails for embedded \n:s. Imnsho, awk needs csv (and json, etc) builtin, preferable as a plugin architecture. But then, why not just use the Perl superset?
        mtdewcmu 13y ago
mtdewcmu 13y ago
No question you can do more with Perl. But I wonder if you've checked out more recent versions of gawk. Modern gawk is a better language than the awk of 20 years ago, so there's less reason to dump it and use Perl.
planckscnst 13y ago
I have one situation where bash + sed is preferable: embedded devices where storage is a primary cost component. Having a shell on the device is somewhat necessary for development. It so happens that the shell is also a decent language interpreter, so adding another interpreter doesn't always justify the cost. I eventually got pretty good at doing both functional and OO-style programming (each where appropriate, and straight-up imperative scripts in many other places) in POSIX-compliant shell.
billjings 13y ago
Author here. You're right that awk is a limited tool. That's exactly why I wanted to write about it. It just wouldn't be possible to make a comprehensive case for perl in a blog post this short, because perl does much more than awk does. And that's kind of why I like awk as much as I do, in spite of its limitations. There's so little to the language, it's of practical use, and as a precursor to tools like perl, python, and ruby, it's historically interesting, too. And I know a lot of people who like to learn neat, useful things who don't know anything about awk. Thanks for reading, btw - I think perl is another older tool that could stand for some evangelizing. I don't write perl, though, so I'm not the guy to do it.

by tux1968 13y ago

cstross 13y ago

Well, considering the original remit for perl (back in the pre-Perl 5, pre-OOP, pre-CPAN days) was to glue sed and awk together with a proper programming language and full access to the POSIX library, that shouldn't be a surprise. Aren't a2p and s2p still part of the standard Perl distribution? (Haven't looked: stopped driving Perl for a living more than ten years ago.) If there's one argument against using perl in place of these other tools it's simply the cognitive overhead of learning the extra stuff perl brings to the table on top of bash, sed and awk. On the other hand, picking up non-OO old-school perl if you're already a proficient shell scripter should be a day's work ... the important thing is knowing when to switch tools.

ivanhoe 13y ago

Perl is great, but today you can just as well (or easier) use ruby, python or even php for scripting one-liners and small tasks. However awk and sed are still fantastic option when you need just a quick processing on a command line. Pipe it in, do some transformations or replacements, pipe it out. Same goes for other unix power-tools like sort, unique, wc, head, tail, split, etc., they are all still very useful...

tux1968 13y ago
Well, I have close to 30 years Unix experience so you don't have to convince me of the power of a pipeline. But I do have to disagree with your affection for awk and sed even though in the past I have used them both extensively. The advantage of learning perl is that you can replace both utilities in your piped command line and only have to remember one syntax. And it integrates just as well as sed and awk do in any pipeline. But quite often, it will obviate the need for piping through other utilities. Perl is much closer to sed and awk than any of the other scripting languages you mention. As noted in other posts, you can even automatically convert awk scripts to perl. And it doesn't take much effort to convert sed scripts to perl where you can leverage more powerful pattern matching and procedural facilities to boot. If you already know sed and awk, by all means keep using them, they're fine. But if you're new to the Unix command line, you'll get the most bang for your buck by learning perl instead of awk and sed.
- ghshephard 13y ago
  As a sysadmin (not web/app or systems programmer) I spent about 7 years using Perl (wrote my company's HR <-> LDAP synchronization system in it) until Python came along (for me, around 2003), nowadays for one or two liners, it's sed/awk - the simplicity of awk '{print $1,$2}', awk '{tot+=$2; c++}END{print tot/c}' has just worked it's way into my fingers, I don't even have to think about sed 's/^.client="$.$",dev.*/\1/' foo.txt - I just look at the log file and the sed expression just appears at my fingertips. Anything more complex than that (unless it's a really obvious fit for awk) I switch over to Python. Its simple data model for things like HoA or AoH just imposes less cognitive overhead on me, and it's rich assembly of built in libraries make me productive from the get-go. This sort of thing would be incredibly difficult in awk - not sure if I would have to go to CPAN with perl - but I just assumed (and was correct) that python would have it built in. import json f=open("bus-stops.json") j=json.load(f) for a in j: print a['no'],",",a['lat'],",",a['lng'],",",a['name'] A simple "import json", "help(json)" at the Python command line, and 2 minutes later I was done. Also - I'm able to understand my code the next day - something I was never able to do with Perl, but for some reason I can with Python. I probably spend 90% of my time in sed/awk, and 10% of my time in Python. Haven't touched Perl in 10 years - not because it isn't an awesome language (it really is) - it's just that I have room in my head for one full blown language at a time, and Python replaced Perl for me.
  - tux1968 13y ago
    Yeah, i have no problem with python or any other language. Use what works for you. But perl does an acceptable job on every one of your examples.. perl -naE 'say $F[0], $F[1]' perl -nae '$tot+=$F[1]; $c++;END{print $tot/$c}' perl -ape 's/^.client="(.)",dev.*/\1/' foo.txt And finally: use JSON; use IO::All; use feature 'say'; $f = io('bus-stops.json')->all; $j = decode_json($f); $,=','; for (@$j) { say @$_{'no','lat','lng','name'}; } I still think newbies would get much more value out of learning perl basics than spending any time on sed/awk intricacies. But to each his own.
  - jnazario 13y ago
    awk tip: in your averaging example, the "c++" and reference to c in the END block can be replaced by the built-in var NR (for number of records). awk's builtins are useful.
berntb 13y ago
Good luck doing one liners with Python, as long as it use wspace indentation for e.g. loops and don't support {}:s as an option. :-( I'd argue with most others, Perl is the single most useful command line tool. Not the only one, of course. But afaik, you can't e.g. load a JSON lib in awk as part of a pipeline. (I deserialize dumped data structures multiple times a week in [ad hoc testing with] pipelined cmds.) Imho, if you know Ruby or PHP as the back of your hand, don't learn another scripting language for command line use. Learn some completely different language for some other use, instead.
- mtdewcmu 13y ago
  No, you can't work with JSON very easily in awk. Any kind of hierarchical format like JSON and XML will give you a headache in awk, and CSV can be difficult as well.
  - bsg75 13y ago
    While not as complete as a full CSV parsing lib, finding this made working with CSV in awk much easier: http://www.gnu.org/software/gawk/manual/html_node/Splitting-By-Content.html http://www.gnu.org/software/gawk/manual/html_node/Splitting-... gawk -vFPAT='[^,]*|"[^"]*"' http://stackoverflow.com/questions/4205431/parse-a-csv-using-awk-and-ignoring-commas-inside-a-field/17287068#17287068 http://stackoverflow.com/questions/4205431/parse-a-csv-using...
    - berntb 13y ago
      That regexp fails with fields containing '"':s, but I guess you can grep for embedded double quotes ("") first. Are there multiple variants of coding '"' in CSV fields? I don't know -- but some people who do know are those who write the CSV libs I use! Edit: And as your link notes, it fails for embedded \n:s. Imnsho, awk needs csv (and json, etc) builtin, preferable as a plugin architecture. But then, why not just use the Perl superset?
      - mtdewcmu 13y ago

mtdewcmu 13y ago

No question you can do more with Perl. But I wonder if you've checked out more recent versions of gawk. Modern gawk is a better language than the awk of 20 years ago, so there's less reason to dump it and use Perl.

planckscnst 13y ago

I have one situation where bash + sed is preferable: embedded devices where storage is a primary cost component. Having a shell on the device is somewhat necessary for development. It so happens that the shell is also a decent language interpreter, so adding another interpreter doesn't always justify the cost. I eventually got pretty good at doing both functional and OO-style programming (each where appropriate, and straight-up imperative scripts in many other places) in POSIX-compliant shell.

billjings 13y ago

Author here. You're right that awk is a limited tool. That's exactly why I wanted to write about it. It just wouldn't be possible to make a comprehensive case for perl in a blog post this short, because perl does much more than awk does. And that's kind of why I like awk as much as I do, in spite of its limitations. There's so little to the language, it's of practical use, and as a precursor to tools like perl, python, and ruby, it's historically interesting, too. And I know a lot of people who like to learn neat, useful things who don't know anything about awk. Thanks for reading, btw - I think perl is another older tool that could stand for some evangelizing. I don't write perl, though, so I'm not the guy to do it.