4 ms·

I was wondering how awk work internally. Does it compile the script then run it ? Is it bytecode or lower level ? A finite state machine ?

CodeArtisan 7y ago
With GNU awk, scripts are compiled to bytecode then interpreted with a big switch/case loop. edit: https://git.savannah.gnu.org/cgit/gawk.git/tree/interpret.h https://git.savannah.gnu.org/cgit/gawk.git/tree/interpret.h
chasil 7y ago
The mawk version of the language will output C, if I remember correctly. It is the fastest AWK, and supports some of the GNU extensions. "The One True AWK" from Brian Kernighan (that is still the system AWK in OpenBSD) switched from a yacc implementation to a custom parser sometime within the last decade (fairly recently). Busybox also has an awk; I'm not sure what they do. GNU awk is elsewhere reported to be interpreted bytecode.
- snaky 7y ago
  Dennis Ritchie about yacc history details > In some ways the interesting thing is that the parser (probably for B, couldn't have been C based on radiocarbon dating evidence) was tiny and dead simple using recursive descent for most parts, a precedence table for expressions. But out of the intellectual culture-meets-culture encounter, an enduring tool was created. https://yarchive.net/comp/handwritten_parse_tables.html https://yarchive.net/comp/handwritten_parse_tables.html
- fanf2 7y ago
  Looks like OpenBSD awk still uses yacc, like other copies of bwk‘s one true awk https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/awk/#dirlist https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/awk/#d...
benhoyt 7y ago
They parse the script to a parse tree (abstract syntax tree) and then either interpret that directly (tree-walking interpreter) or compile to bytecode and then execute that. The original awk ("one true awk") uses a simple tree-walking interpreter, as does my own GoAWK implementation. gawk and mawk are slightly faster and compile to bytecode first. If you're interested, you can read more about how GoAWK works and performs here: https://benhoyt.com/writings/goawk/ https://benhoyt.com/writings/goawk/
- srean 7y ago
  Did gawk change to bytecode interpreter recently ? As far as I can recall it used to be an AST walker. Wonder if I am misremembering things. Mawk used tobe more than 'slightly' faster than gawk. Maybe the recent change to bytecode have brought their perf characteristics closer.
- julienfr112 7y ago
  Nice work !

CodeArtisan 7y ago

With GNU awk, scripts are compiled to bytecode then interpreted with a big switch/case loop. edit: https://git.savannah.gnu.org/cgit/gawk.git/tree/interpret.h https://git.savannah.gnu.org/cgit/gawk.git/tree/interpret.h

chasil 7y ago

The mawk version of the language will output C, if I remember correctly. It is the fastest AWK, and supports some of the GNU extensions. "The One True AWK" from Brian Kernighan (that is still the system AWK in OpenBSD) switched from a yacc implementation to a custom parser sometime within the last decade (fairly recently). Busybox also has an awk; I'm not sure what they do. GNU awk is elsewhere reported to be interpreted bytecode.

snaky 7y ago
Dennis Ritchie about yacc history details > In some ways the interesting thing is that the parser (probably for B, couldn't have been C based on radiocarbon dating evidence) was tiny and dead simple using recursive descent for most parts, a precedence table for expressions. But out of the intellectual culture-meets-culture encounter, an enduring tool was created. https://yarchive.net/comp/handwritten_parse_tables.html https://yarchive.net/comp/handwritten_parse_tables.html
fanf2 7y ago
Looks like OpenBSD awk still uses yacc, like other copies of bwk‘s one true awk https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/awk/#dirlist https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/awk/#d...

benhoyt 7y ago

They parse the script to a parse tree (abstract syntax tree) and then either interpret that directly (tree-walking interpreter) or compile to bytecode and then execute that. The original awk ("one true awk") uses a simple tree-walking interpreter, as does my own GoAWK implementation. gawk and mawk are slightly faster and compile to bytecode first. If you're interested, you can read more about how GoAWK works and performs here: https://benhoyt.com/writings/goawk/ https://benhoyt.com/writings/goawk/

srean 7y ago
Did gawk change to bytecode interpreter recently ? As far as I can recall it used to be an AST walker. Wonder if I am misremembering things. Mawk used tobe more than 'slightly' faster than gawk. Maybe the recent change to bytecode have brought their perf characteristics closer.
julienfr112 7y ago
Nice work !