3 ms·

As far as I remember that talk, the parsing problems come down to two points: - Shell aliases are grammar-defying macros in basically the same way as C preproc

As far as I remember that talk, the parsing problems come down to two points: - Shell aliases are grammar-defying macros in basically the same way as C preprocessors macros are but unlike those are not separable from the actual execution, which, yes, they’re a misfeature that defies static analysis, I don’t know why people seem to prefer aliases to shell functions so much; - The "modern" $(... $(...) ...) syntax, unlike the "legacy" (per shellcheck) `... \`...\` ...` syntax, requires invoking the parser from the lexer, which, also yes, but as far as I know that’s a problem that every language with general expression interpolation must face—Python’s f-strings tried to sidestep the problem, but I’m not sure the resulting restricted language with weird escaping rules was worth it. For a talk that was supposed to be about problems encountered while building a parser, I was surprised by how much of a non-event both of the problems were.

chasil 4y ago
This does not change the ambiguity of the language, nor does it remove the need for a prelexer. Quoting the talk, very quickly... Which command outputs \\? echo "\\\" echo "\\\\" echo "\\\\\\" You may see this as a non-event, but the rest of us are not quite comfortable with this.
- Karellen 4y ago
  How is that different from backslash-escaped strings in C, Java, Javascript, Python, and... uh, a whole bunch of other common languages? Which of those gives you a string with two backslashes in those other languages? Exactly the same one as (ba)sh.
  - chasil 4y ago
    Let's have a more concrete example of ambiguity. "${*##*[ ]}" is a valid POSIX approach to getting the last parameter but it doesn't work in ksh or bash. Those shells apply the pattern to each parameter in turn when used on '*' or '@'. That text is from a developer who has profound depth. A portable method to extract the last argument is: for last do : done This ambiguity and subtlety is not good.
- mananaysiempre 4y ago
  Counting backslashes is unpleasant in any language, and Bourne shell excluding `...` does not seem worse than any other. Pascal-style quote doubling is the only reasonable alternative I know of, and except for occupying one character instead of two it’s not all that much better. (Also, trick question, because the first line, with its odd number of backslashes, is a syntax error if given on its own.) More importantly, the difficulties humans have with the language are different from the difficulties computers do, and I was mostly thinking about the latter, what with the mention of LR parsing and everything.
  - H4ZB7 4y ago
    nope, escaping is definitely worse in bash (and all shells) than real languages.
xigoi 4y ago
What's the point of having a separate lexing and parsing step anyway?
- mananaysiempre 4y ago
  Two reasons, one questionably relevant and one difficult to formalize. The questionably relevant reason is that all well-known classes of declarative syntax specifications that are easy to parse don’t really compose very well, in any direction; the exception being PEGs, which do compose, but in nonintuitive ways, and are also kind of difficult to parse. So if you want a generated parser, chaining a regular-language-based lexer and an LL- or LR-based parser lets you tackle a much wider class of languages than the second step alone. Of course, hardly any serious language implementations use generated parsers these days, and hand-rolling a scannerless parser is not that much more difficult as far as the mechanics go. But then, the difficult-to-formalize reason is that people do actually think of languages that way, and separating phases gets you more localized and more understandable errors. This might be an artifact of how language specs are written, but I don’t think so: good frontends will occasionally split the process into even more phases, first parsing a looser syntax than specified and then tightening it up with semantic checks. For example, ISO C prohibits (a + b = c) at the grammar level, by restricting which expression productions are allowed to the left of an assignment, but a good compiler will parse this successfully and only afterwards tell you that (a + b) is not a thing you can assign to. In this connection, the "skeleton syntax tree" idea as used in Dylan[1] seems promising, but I don’t think I’ve seen it developed further. [1] Bachrach, Playford, "D-expressions: Lisp power, Dylan style", https://people.csail.mit.edu/jrb/Projects/dexprs.pdf https://people.csail.mit.edu/jrb/Projects/dexprs.pdf
  - marssaxman 4y ago
    Thanks for the reference to D-expressions; that was an interesting read.

chasil 4y ago

This does not change the ambiguity of the language, nor does it remove the need for a prelexer. Quoting the talk, very quickly... Which command outputs \\? echo "\\\" echo "\\\\" echo "\\\\\\" You may see this as a non-event, but the rest of us are not quite comfortable with this.

Karellen 4y ago
How is that different from backslash-escaped strings in C, Java, Javascript, Python, and... uh, a whole bunch of other common languages? Which of those gives you a string with two backslashes in those other languages? Exactly the same one as (ba)sh.
- chasil 4y ago
  Let's have a more concrete example of ambiguity. "${*##*[ ]}" is a valid POSIX approach to getting the last parameter but it doesn't work in ksh or bash. Those shells apply the pattern to each parameter in turn when used on '*' or '@'. That text is from a developer who has profound depth. A portable method to extract the last argument is: for last do : done This ambiguity and subtlety is not good.
mananaysiempre 4y ago
Counting backslashes is unpleasant in any language, and Bourne shell excluding `...` does not seem worse than any other. Pascal-style quote doubling is the only reasonable alternative I know of, and except for occupying one character instead of two it’s not all that much better. (Also, trick question, because the first line, with its odd number of backslashes, is a syntax error if given on its own.) More importantly, the difficulties humans have with the language are different from the difficulties computers do, and I was mostly thinking about the latter, what with the mention of LR parsing and everything.
- H4ZB7 4y ago
  nope, escaping is definitely worse in bash (and all shells) than real languages.

xigoi 4y ago

What's the point of having a separate lexing and parsing step anyway?

mananaysiempre 4y ago
Two reasons, one questionably relevant and one difficult to formalize. The questionably relevant reason is that all well-known classes of declarative syntax specifications that are easy to parse don’t really compose very well, in any direction; the exception being PEGs, which do compose, but in nonintuitive ways, and are also kind of difficult to parse. So if you want a generated parser, chaining a regular-language-based lexer and an LL- or LR-based parser lets you tackle a much wider class of languages than the second step alone. Of course, hardly any serious language implementations use generated parsers these days, and hand-rolling a scannerless parser is not that much more difficult as far as the mechanics go. But then, the difficult-to-formalize reason is that people do actually think of languages that way, and separating phases gets you more localized and more understandable errors. This might be an artifact of how language specs are written, but I don’t think so: good frontends will occasionally split the process into even more phases, first parsing a looser syntax than specified and then tightening it up with semantic checks. For example, ISO C prohibits (a + b = c) at the grammar level, by restricting which expression productions are allowed to the left of an assignment, but a good compiler will parse this successfully and only afterwards tell you that (a + b) is not a thing you can assign to. In this connection, the "skeleton syntax tree" idea as used in Dylan[1] seems promising, but I don’t think I’ve seen it developed further. [1] Bachrach, Playford, "D-expressions: Lisp power, Dylan style", https://people.csail.mit.edu/jrb/Projects/dexprs.pdf https://people.csail.mit.edu/jrb/Projects/dexprs.pdf
- marssaxman 4y ago
  Thanks for the reference to D-expressions; that was an interesting read.