5 ms·
You get the gist of the benefits—to get to a point where you can "implement JavaScript-style RegEx literals" without too much effort by removing some barriers t
by cxr 3y ago
You get the gist of the benefits—to get to a point where you can "implement JavaScript-style RegEx literals" without too much effort by removing some barriers to implementation—but some of the conceptual specifics in your understanding are off.
> I think what your saying is the following?:¶ The scanner has a "(/" token which triggers dropping into a different scanner/parser for RegExes
Not quite. First, you don't have separate `(/` and `/)` tokens. The `(/` character sequence is the start of a single pattern literal (the same way that if you're parsing JSON, the opening double quote is part of the string literal). What it does is serve as an unambiguous symbol that yes, what we are seeing here is a literal for a PCRE-style pattern. Secondly, you don't have to have a separate scanner. That's sort of the point—supporting "eggular expressions" is easy because tokenizing them is drastically simpler than the hacks regarding context-sensitivity that you must weave into both your parser and scanner (and complicate the scanner–parser contract) if you want conventional unparenthesized literals. Adding support for the parenthesized form to your lexer amounts to just a few lines of code, and not only that, but very simple, very boring code of the same sort that you have elsewhere—easier than the code to support numeric literals, even—whereas the same is distinctly not true if they are allowed to be unparenthesized.
(NB: I mentioned 80/20 before. These "eggexes" are also the good kind of partial implementation: a pure case of forward compatibility—if you support the parenthesized form now, there's nothing stopping you from relaxing the paren constraint in the future and announcing that you now support unparenthesized literals.)
> I don't love the user experience of that, but maybe that's just an aesthetic opinion.
Yeah, aesthetic is right. I actually prefer them parenthesized, anyway, for entirely unrelated reasons. Similar constructions like "foo".bar and [].baz, while technically permitted in the grammars of several languages, raise my hackles. It's no less true concerning literals used for pattern matching. Parens need to be inserted on the lhs of the dot for anyone who wants my approval in code review. As it happens, though, there are benefits to readability on multiple levels (i.e. for both humans and machines).
- kerkeslager 3y ago> Secondly, you don't have to have a separate scanner. That's sort of the point—supporting "eggular expressions" is easy because tokenizing them is drastically simpler than the hacks regarding context-sensitivity that you must weave into both your parser and scanner (and complicate the scanner–parser contract) if you want conventional unparenthesized literals. Hm, I'm not sure I understand this bit. It seems bad to use the same scanner for the tokens inside your regular expression as for the rest of your language. Two examples: 1. In my langauge, "3.14" is a token, but when parsing an eggex you'd want that to be at least 3 tokens because the "." is typically your "match one of any character" token. 2. My language doesn't have a "?" operator, but if we're using the same scanner for eggexes as the rest of the language, I'd have to include a token for that to support "zero or one instances" in the eggexes. Sure, your main scanner can (and probably should) just grab everything between "(/" and "/)" as one token, but there's no avoiding going over the contents of the eggex with a scanner. I'm not saying you have to scan it during the regular scanning--that's probably not a good idea--but you're going to have scan it at some point in order to compile the eggex.