3 ms·

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell th

Very nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later that day, I came across another task which seemed too complex for an awk one-liner so I took two lines of R and was surprised when R was done within 5 seconds on the same data set. I was happy because I found a faster tool than the one I had, but the lesson is, that just because you use a proven tool like awk, doesn't mean there aren't any better tools. Find out what works best for you.

wodenokoto 7y ago
What kind of task was it? I’d imagine loading a csv-like file and doing some somewhat heavy calculations, R would win, but when I think of AWK problems I think of text manipulation, and when I think of manipulating text I do not think of R Disclaimer: I don’t actually write any AWK, but learning it is on my bucket list.
- arendtio 7y ago
  The tasks were different. AWK -> calculate the difference between every two lines: awk -F ',' 'NR!=1{printf "%.0f\n", $1-ll}NR==1{print ""}{ll=$1}' R -> Calculate the min, median, mean, and max: d<-scan("stdin", quiet=TRUE) cat(min(d), max(d), median(d), mean(d), sep=" ") AFAIK, calculating the difference to its previous line for every line should be faster for data that is not sorted. I didn't try to write that one in R though. I am sure there ways to make both things faster, I was just surprised as I didn't expect R to be faster with those two naive implementations.
tannhaeuser 7y ago
awk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of which, I've noticed gawk on recent Ubuntus (19.10) appears broken and/or has regexp size limits breaking awk code with large regexpes (but haven't checked yet thorougly). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...
- arendtio 7y ago
  Arch Linux $ awk --version GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
- arendtio 7y ago
  Since others were suggesting mawk too, I just installed it and it was indeed faster than GNU awk: The task took about 7 seconds. Interesting enough: LANG=C increased the performance of GNU awk a bit, but actually decreased the mawk performance.
dev_tty01 7y ago
Was it a faster tool? You had a need, you tossed awk at it and accomplished your goal. Did the 20 seconds vs. 5 seconds matter? The time you spent deciding what to do and how to do it took more time. For a one-off, grab some data type of task, the tool that you know well will almost always be the fastest because you get to the end result the quickest. I agree with the comment below. If you haven't already, try mawk instead of awk. It is often many times faster than awk (and other solutions).
ratsmack 7y ago
I have never found anything faster then mawk. GNU awk (gawk) is orders of magnitude slower. For a proper comparison you would have to note which implementation of awk you are using.

wodenokoto 7y ago

What kind of task was it? I’d imagine loading a csv-like file and doing some somewhat heavy calculations, R would win, but when I think of AWK problems I think of text manipulation, and when I think of manipulating text I do not think of R Disclaimer: I don’t actually write any AWK, but learning it is on my bucket list.

arendtio 7y ago
The tasks were different. AWK -> calculate the difference between every two lines: awk -F ',' 'NR!=1{printf "%.0f\n", $1-ll}NR==1{print ""}{ll=$1}' R -> Calculate the min, median, mean, and max: d<-scan("stdin", quiet=TRUE) cat(min(d), max(d), median(d), mean(d), sep=" ") AFAIK, calculating the difference to its previous line for every line should be faster for data that is not sorted. I didn't try to write that one in R though. I am sure there ways to make both things faster, I was just surprised as I didn't expect R to be faster with those two naive implementations.

tannhaeuser 7y ago

awk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of which, I've noticed gawk on recent Ubuntus (19.10) appears broken and/or has regexp size limits breaking awk code with large regexpes (but haven't checked yet thorougly). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...

arendtio 7y ago
Arch Linux $ awk --version GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
arendtio 7y ago
Since others were suggesting mawk too, I just installed it and it was indeed faster than GNU awk: The task took about 7 seconds. Interesting enough: LANG=C increased the performance of GNU awk a bit, but actually decreased the mawk performance.

dev_tty01 7y ago

Was it a faster tool? You had a need, you tossed awk at it and accomplished your goal. Did the 20 seconds vs. 5 seconds matter? The time you spent deciding what to do and how to do it took more time. For a one-off, grab some data type of task, the tool that you know well will almost always be the fastest because you get to the end result the quickest. I agree with the comment below. If you haven't already, try mawk instead of awk. It is often many times faster than awk (and other solutions).

ratsmack 7y ago

I have never found anything faster then mawk. GNU awk (gawk) is orders of magnitude slower. For a proper comparison you would have to note which implementation of awk you are using.