3 ms·

Shells have somewhat unusual parsing requirements. For example "if" is a keyword when used as `if echo` but not `echo if`. So you either need to implement the

Shells have somewhat unusual parsing requirements. For example "if" is a keyword when used as `if echo` but not `echo if`. So you either need to implement the lexer hack, or have a "string" token type which is disambiguated by the parser (which is what fish-shell does). https://en.wikipedia.org/wiki/Lexer_hack https://en.wikipedia.org/wiki/Lexer_hack

radiospiel 2y ago
unless i miss something this should not be an issue. the lexer could parse if as an IF token, and the parser could treat tags as STRING || IF ( || other keywords… )
- duskwuff 2y ago
  That seems like it'd get really awkward pretty quickly. "if" isn't unique in this regard; there are about a hundred shell builtins, and all of them can be used as an argument to a command. (For example, "echo then complete command while true history" is a valid shell command consisting entirely of names of builtins, and the only keyword in it is the leading "echo".)
- deathanatos 2y ago
  You'd have to `|| EVERY_KEYWORD_IN_LANG`, and then if you ever add a keyword, now you're updating that list there, and anywhere else you've used it. As the "Lexer hack" Wiki page says, this is only a problem if you're lexing in the first place. If you just parse the grammar, this isn't a problem.
- hnlmorg 2y ago
  The problem lies with shells extensive usage of barewords. If you could eliminate the requirement for any bareword to be treated as a string then parsing shell code would then become much simpler...but also few people would want to use it because nobody wants to write the following in their interactive shell: git "commit" "-am" "message" ls "-l" etc
throwaway2016a 2y ago
That's no problem in many modern lexers as they usually have a "state" so when you encounter "echo" you can switch to a new state and that state may have different token parsing rules. So "if" in the "echo" state could be a string literal whereas it may be a keyword in the initial state. Lex/Flex takes care of that mostly for you which is one of the benefits of using a well worn lexer generator and not rolling your own.

radiospiel 2y ago

unless i miss something this should not be an issue. the lexer could parse if as an IF token, and the parser could treat tags as STRING || IF ( || other keywords… )

duskwuff 2y ago
That seems like it'd get really awkward pretty quickly. "if" isn't unique in this regard; there are about a hundred shell builtins, and all of them can be used as an argument to a command. (For example, "echo then complete command while true history" is a valid shell command consisting entirely of names of builtins, and the only keyword in it is the leading "echo".)
deathanatos 2y ago
You'd have to `|| EVERY_KEYWORD_IN_LANG`, and then if you ever add a keyword, now you're updating that list there, and anywhere else you've used it. As the "Lexer hack" Wiki page says, this is only a problem if you're lexing in the first place. If you just parse the grammar, this isn't a problem.
hnlmorg 2y ago
The problem lies with shells extensive usage of barewords. If you could eliminate the requirement for any bareword to be treated as a string then parsing shell code would then become much simpler...but also few people would want to use it because nobody wants to write the following in their interactive shell: git "commit" "-am" "message" ls "-l" etc

throwaway2016a 2y ago

That's no problem in many modern lexers as they usually have a "state" so when you encounter "echo" you can switch to a new state and that state may have different token parsing rules. So "if" in the "echo" state could be a string literal whereas it may be a keyword in the initial state. Lex/Flex takes care of that mostly for you which is one of the benefits of using a well worn lexer generator and not rolling your own.