7 ms·

I have, and read, a book on awk (the AWK programming language, by guess-the-initials) and didn't see the point. If I need regexes I have emacs. For CSVs I use e

I have, and read, a book on awk (the AWK programming language, by guess-the-initials) and didn't see the point. If I need regexes I have emacs. For CSVs I use excel or LibreOffice. It has been a while so could someone remind me why awk is so fab? Perhaps it's just I've never met the right use case to make me fall in love. Sorry if it's a dumb question. TIA

tgv 7y ago
AWK's primary use case is processing a file line by line. It offers a few well thought out mechanisms to do so, allowing you to script relatively complex operations in just a few lines. At the same time, and acknowledging it's not pretty, it's powerful enough to run an http server (not kidding: https://github.com/kevin-albert/awkserver https://github.com/kevin-albert/awkserver). So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend. Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.
the_jeremy 7y ago
excel can't handle millions of lines. Every job (only 2, but still) I've had, I've had a problem at least once where I'm given millions of lines and needed to parse them for something. That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.
- tempguy9999 7y ago
  Yes, this was my feeling too, why not just use python and save the effort of learning another language? Perhaps because you can't assume python will be present in a basic *nix install.
fian 7y ago
Awk predates Excel (and therefore LibreOffice), so the original point was that...there was no GUI spreadsheet application to use instead. 32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded. Excel requires all the data to be in memory. Awk processes line by line so requires less memory. Awk scripts can easily be incorporated in a chain of other commands. Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files. Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems. Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk. Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.
ekianjo 7y ago
For very large CSV files you cant use excel or libre office because they cannot even open the files. Instead you just use bash tools like awk, sed, grep to get what you need. Even for moderately large CSV files these tools are miles faster than GUI applications.

tgv 7y ago

AWK's primary use case is processing a file line by line. It offers a few well thought out mechanisms to do so, allowing you to script relatively complex operations in just a few lines. At the same time, and acknowledging it's not pretty, it's powerful enough to run an http server (not kidding: https://github.com/kevin-albert/awkserver https://github.com/kevin-albert/awkserver). So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend. Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.

the_jeremy 7y ago

excel can't handle millions of lines. Every job (only 2, but still) I've had, I've had a problem at least once where I'm given millions of lines and needed to parse them for something. That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.

tempguy9999 7y ago
Yes, this was my feeling too, why not just use python and save the effort of learning another language? Perhaps because you can't assume python will be present in a basic *nix install.

fian 7y ago

Awk predates Excel (and therefore LibreOffice), so the original point was that...there was no GUI spreadsheet application to use instead. 32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded. Excel requires all the data to be in memory. Awk processes line by line so requires less memory. Awk scripts can easily be incorporated in a chain of other commands. Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files. Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems. Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk. Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.

ekianjo 7y ago

For very large CSV files you cant use excel or libre office because they cannot even open the files. Instead you just use bash tools like awk, sed, grep to get what you need. Even for moderately large CSV files these tools are miles faster than GUI applications.