3 ms·
Thanks for that. Indeed it makes sense. I would enjoy reading blog posts about your lexer. What through me is that the BNF presented in the POSIX standard tr
by aidenn0 10y ago
Thanks for that. Indeed it makes sense. I would enjoy reading blog posts about your lexer.
What through me is that the BNF presented in the POSIX standard treats words as individual tokens and that's obviously not something you want to do only in a lexer.
- chubot 10y agoYes, so bash is unique in that there is a two level structure. Tokens turn are parsed into words. And then words are used as "tokens" for the "command sublanguage", the grammar for which is given in POSIX. I mention that in a few places -- there are 4 sublanguages, arith/bool/word/command, but POSIX only covers the command sublanguage. http://www.oilshell.org/blog/2016/10/26.html http://www.oilshell.org/blog/2016/10/26.html This is one of the reasons parser generators as "frameworks" don't work for shell. I have a diagram in mind for the parser architecture which I think will be illuminating. The command language uses the words as "tokens", but they're also mutually recursive, because an entire subprogram can fit inside a word/string, like "prefix $(my-func() { echo hi; }; my-func) suffix" That is defining a function and calling it inside a string.