4 ms·

> 1. Intent is easily lost on read. In actual code, it is of course attached to some definition like `type: TOK_STRING`. > 2. Balanced pairs allowing an escap

> 1. Intent is easily lost on read. In actual code, it is of course attached to some definition like `type: TOK_STRING`. > 2. Balanced pairs allowing an escaped terminator are not “regular language” Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular. /"(?:\\.|.)*?"/ This makes it impossible to include a " in the string; the original regexp works because it seeks the longest match. At this point you might as well have written /".*?"/ or equivalently for a greedy match, /"[^"]*"/ > I’m certain this would be an XSS vulnerability. This feels like a big leap for something I didn't say was even involved in web development. I see ways that a failure to handle an invalid string (and this will match invalid string in languages that have additional escape forms with stricter requirements e.g. `\u1234`) could be chained with other bugs to create an XSS, but it would be difficult to ascribe the XSS to this part. And extension to handle most such things wouldn't make the language non-regular. (p.s. > I’ll include slash delimiters to remove ambiguity. The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".)

morelisp 6y ago
This is slightly wrong, depending on if your regexp engine requires non-greedy matches to find the shortest possible match or not. (I rarely use non-greedy matches because of this, among other reasons.) /"(?:\\.|.)*?"/ May either stop at the first ", even if escaped (the \ will match the . case), or it will continue to the last " even if escaped (for the same reason). If anything, you've introduced a vulnerability by breaking the escape handling.
- eyelidlessness 6y ago
  I’m open to the possibility this will vary based on engine, but I don’t think it should. The pattern specifies: 1. Match a quote. 2. Match the shortest sequence of: A) any character preceded by a backslash, or B) any thus far unmatched character, until 3. Matching the first unmatched quote. The 2.A case should match before laziness kicks in. The only case when it shouldn’t is if the backslash has already been consumed by a previous escape.
  - morelisp 6y ago
    On the main issue of your regexp being wrong: You can feed your regexp into any engine and see it will happy match "\" which it should not. As for the issue of varying between engines: The classic deviation between POSIX and non-POSIX regexp is whether you pick leftmost-longest and tiebreak by alternate order, or leftmost first alternate. (POSIX is the correct answer because there are benefits to commutative alternation.) Non-greedy regexp has a similar issue for whether it should prefer shortest or first alternate. (Proper thinking about regexes doesn't involve temporal relations like "before" or "thus far" or whether something is "consumed" - even in engines that don't implement leftmost-longest this can only confuse.) Engines that take alternation order into account will also have it match the complete string of: "\"" Probably to the confusion of anyone who read that non-greedy operators will choose the shortest match. tl;dr: Don't use non-greedy operators. Don't think about regular language matching in temporal or procedural terms. Don't try to rewrite regexps without test cases.
eyelidlessness 6y ago
> In actual code, it is of course attached to some definition like `type: TOK_STRING`. Yes, naming helps. I was mostly sympathizing with the people whose brains explode at the sight of a regex. It’s surprising how much time it took me to be sure I understood your simple pattern. > Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular. This is a good correction, thank you. I was thinking of cases where balanced pairs allow both escaped delimiters and nested pairs. > This feels like a big leap for something I didn't say was even involved in web development. I hope it’s clear I didn’t intend to make that leap, and was just generally reflecting on the ways simple patterns can go awry. That’s why I said “for example” and “would”. If someone were to take this pattern or any of the variants and use it in a context that writes innerHTML, it would be vulnerable to XSS by failing to account for multi-character tokens like " in its escaping. > The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".) I understand that many languages don’t use slash delimiters. But it’s quite common to use them for regex where a language context isn’t assumed. My point wasn’t to impose slashes on your example but to make it explicitly knowable, in words, whether the quotes were intended as part of the pattern. Not knowing that was the primary reason I wasn’t sure I understood the pattern’s intent.

morelisp 6y ago

This is slightly wrong, depending on if your regexp engine requires non-greedy matches to find the shortest possible match or not. (I rarely use non-greedy matches because of this, among other reasons.) /"(?:\\.|.)*?"/ May either stop at the first ", even if escaped (the \ will match the . case), or it will continue to the last " even if escaped (for the same reason). If anything, you've introduced a vulnerability by breaking the escape handling.

eyelidlessness 6y ago
I’m open to the possibility this will vary based on engine, but I don’t think it should. The pattern specifies: 1. Match a quote. 2. Match the shortest sequence of: A) any character preceded by a backslash, or B) any thus far unmatched character, until 3. Matching the first unmatched quote. The 2.A case should match before laziness kicks in. The only case when it shouldn’t is if the backslash has already been consumed by a previous escape.
- morelisp 6y ago
  On the main issue of your regexp being wrong: You can feed your regexp into any engine and see it will happy match "\" which it should not. As for the issue of varying between engines: The classic deviation between POSIX and non-POSIX regexp is whether you pick leftmost-longest and tiebreak by alternate order, or leftmost first alternate. (POSIX is the correct answer because there are benefits to commutative alternation.) Non-greedy regexp has a similar issue for whether it should prefer shortest or first alternate. (Proper thinking about regexes doesn't involve temporal relations like "before" or "thus far" or whether something is "consumed" - even in engines that don't implement leftmost-longest this can only confuse.) Engines that take alternation order into account will also have it match the complete string of: "\"" Probably to the confusion of anyone who read that non-greedy operators will choose the shortest match. tl;dr: Don't use non-greedy operators. Don't think about regular language matching in temporal or procedural terms. Don't try to rewrite regexps without test cases.

eyelidlessness 6y ago

> In actual code, it is of course attached to some definition like `type: TOK_STRING`. Yes, naming helps. I was mostly sympathizing with the people whose brains explode at the sight of a regex. It’s surprising how much time it took me to be sure I understood your simple pattern. > Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular. This is a good correction, thank you. I was thinking of cases where balanced pairs allow both escaped delimiters and nested pairs. > This feels like a big leap for something I didn't say was even involved in web development. I hope it’s clear I didn’t intend to make that leap, and was just generally reflecting on the ways simple patterns can go awry. That’s why I said “for example” and “would”. If someone were to take this pattern or any of the variants and use it in a context that writes innerHTML, it would be vulnerable to XSS by failing to account for multi-character tokens like " in its escaping. > The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".) I understand that many languages don’t use slash delimiters. But it’s quite common to use them for regex where a language context isn’t assumed. My point wasn’t to impose slashes on your example but to make it explicitly knowable, in words, whether the quotes were intended as part of the pattern. Not knowing that was the primary reason I wasn’t sure I understood the pattern’s intent.