10 ms·

A personal common favorite: "(?:[^\\"]|\\.)*"

eyelidlessness 6y ago
Disclaimer: none of what follows is criticism of you, of your contribution, or even of regex. I’m actually tickled by such a terse comment giving me so much to think through. - - - This is my fourth attempt at understanding the intent of this pattern. I’m quite comfortable with regex, so much so I listed it on my resume skills list for years as sort of a glib joke. I think I’ve been overthinking this one. For my benefit I’m gonna describe what I think this matches and ask you to confirm. For others I hope the description helps. This appears to be intended to match a balanced double quote, where any characters between may be anything besides a backslash or double quote, or any character preceded by a backslash. The content between the bounding quotes isn’t captured, which is to say parentheses usually designate a capturing binding like $1 or \1 depending on language/context, but the ?: tells the regex engine you don’t need/want that binding. If I’m understanding all of that correctly, it’s probably very close to what you intended but definitely highlights two of the biggest problems with regex: 1. Intent is easily lost on read. This is trite to say, but I’m pointing it out as someone who’s played regex golf for fun and still spent over an hour getting confident enough to describe what this short pattern appears to do. 2. Balanced pairs allowing an escaped terminator are not “regular language”. It’s possible (and I think probable but I’m not at a device capable to confirm) that this pattern is safe because the balanced pairs are single-character and there’s only one possible escape. But the fact that I can’t say so with confidence is something that would make me cautious using it. All of that said, if I were writing this. I’d try to make it both more clear in intent and a little more predictably resilient. 1. I’ll include slash delimiters to remove ambiguity. /"(?:[^\\"]|\\.)*"/ 2. I’ll express the escape exception first: /"(?:\\.|[^\\"])*"/ 3. I’ll remove the need to check unescaped characters by terminating the match on the first unescaped quote (using ? after a wildcard is non-greedy and stops at the first match; the default is to match to the last): /"(?:\\.|.)*?"/ I think this makes me feel a little more confident with the escaping, but I’m still iffy without testing it. But it does make for a pretty cool emoticon! Final thoughts: - Almost everywhere you could use regex will not match any kind of line breaks. That might be desirable but depends on what you’re trying to do. - Even with the abundance of caution I expressed, this is decidedly unsafe for processing user input. For example I’m certain this would be an XSS vulnerability.
- morelisp 6y ago
  > 1. Intent is easily lost on read. In actual code, it is of course attached to some definition like `type: TOK_STRING`. > 2. Balanced pairs allowing an escaped terminator are not “regular language” Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular. /"(?:\\.|.)*?"/ This makes it impossible to include a " in the string; the original regexp works because it seeks the longest match. At this point you might as well have written /".*?"/ or equivalently for a greedy match, /"[^"]*"/ > I’m certain this would be an XSS vulnerability. This feels like a big leap for something I didn't say was even involved in web development. I see ways that a failure to handle an invalid string (and this will match invalid string in languages that have additional escape forms with stricter requirements e.g. `\u1234`) could be chained with other bugs to create an XSS, but it would be difficult to ascribe the XSS to this part. And extension to handle most such things wouldn't make the language non-regular. (p.s. > I’ll include slash delimiters to remove ambiguity. The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".)
  - morelisp 6y ago
    This is slightly wrong, depending on if your regexp engine requires non-greedy matches to find the shortest possible match or not. (I rarely use non-greedy matches because of this, among other reasons.) /"(?:\\.|.)*?"/ May either stop at the first ", even if escaped (the \ will match the . case), or it will continue to the last " even if escaped (for the same reason). If anything, you've introduced a vulnerability by breaking the escape handling.
    - eyelidlessness 6y ago
      I’m open to the possibility this will vary based on engine, but I don’t think it should. The pattern specifies: 1. Match a quote. 2. Match the shortest sequence of: A) any character preceded by a backslash, or B) any thus far unmatched character, until 3. Matching the first unmatched quote. The 2.A case should match before laziness kicks in. The only case when it shouldn’t is if the backslash has already been consumed by a previous escape.

Disclaimer: none of what follows is criticism of you, of your contribution, or even of regex. I’m actually tickled by such a terse comment giving me so much to think through. - - - This is my fourth attempt at understanding the intent of this pattern. I’m quite comfortable with regex, so much so I listed it on my resume skills list for years as sort of a glib joke. I think I’ve been overthinking this one. For my benefit I’m gonna describe what I think this matches and ask you to confirm. For others I hope the description helps. This appears to be intended to match a balanced double quote, where any characters between may be anything besides a backslash or double quote, or any character preceded by a backslash. The content between the bounding quotes isn’t captured, which is to say parentheses usually designate a capturing binding like $1 or \1 depending on language/context, but the ?: tells the regex engine you don’t need/want that binding. If I’m understanding all of that correctly, it’s probably very close to what you intended but definitely highlights two of the biggest problems with regex: 1. Intent is easily lost on read. This is trite to say, but I’m pointing it out as someone who’s played regex golf for fun and still spent over an hour getting confident enough to describe what this short pattern appears to do. 2. Balanced pairs allowing an escaped terminator are not “regular language”. It’s possible (and I think probable but I’m not at a device capable to confirm) that this pattern is safe because the balanced pairs are single-character and there’s only one possible escape. But the fact that I can’t say so with confidence is something that would make me cautious using it. All of that said, if I were writing this. I’d try to make it both more clear in intent and a little more predictably resilient. 1. I’ll include slash delimiters to remove ambiguity. /"(?:[^\\"]|\\.)*"/ 2. I’ll express the escape exception first: /"(?:\\.|[^\\"])*"/ 3. I’ll remove the need to check unescaped characters by terminating the match on the first unescaped quote (using ? after a wildcard is non-greedy and stops at the first match; the default is to match to the last): /"(?:\\.|.)*?"/ I think this makes me feel a little more confident with the escaping, but I’m still iffy without testing it. But it does make for a pretty cool emoticon! Final thoughts: - Almost everywhere you could use regex will not match any kind of line breaks. That might be desirable but depends on what you’re trying to do. - Even with the abundance of caution I expressed, this is decidedly unsafe for processing user input. For example I’m certain this would be an XSS vulnerability.

morelisp 6y ago
> 1. Intent is easily lost on read. In actual code, it is of course attached to some definition like `type: TOK_STRING`. > 2. Balanced pairs allowing an escaped terminator are not “regular language” Are you confusing this with that e.g. the language of validly parenthesized expressions is not regular? A single string with the option to escape any character including the escape character and terminator is regular. /"(?:\\.|.)*?"/ This makes it impossible to include a " in the string; the original regexp works because it seeks the longest match. At this point you might as well have written /".*?"/ or equivalently for a greedy match, /"[^"]*"/ > I’m certain this would be an XSS vulnerability. This feels like a big leap for something I didn't say was even involved in web development. I see ways that a failure to handle an invalid string (and this will match invalid string in languages that have additional escape forms with stricter requirements e.g. `\u1234`) could be chained with other bugs to create an XSS, but it would be difficult to ascribe the XSS to this part. And extension to handle most such things wouldn't make the language non-regular. (p.s. > I’ll include slash delimiters to remove ambiguity. The two languages I have most often used this in don't use slash delimiters, so I think this has only added ambiguity - now we are considering it embedded in some specific-but-unspecified language's regexp syntax where we might need to consider another level of escaping - compared to letting the regex stand "bare".)
- morelisp 6y ago
  This is slightly wrong, depending on if your regexp engine requires non-greedy matches to find the shortest possible match or not. (I rarely use non-greedy matches because of this, among other reasons.) /"(?:\\.|.)*?"/ May either stop at the first ", even if escaped (the \ will match the . case), or it will continue to the last " even if escaped (for the same reason). If anything, you've introduced a vulnerability by breaking the escape handling.
  - eyelidlessness 6y ago
    I’m open to the possibility this will vary based on engine, but I don’t think it should. The pattern specifies: 1. Match a quote. 2. Match the shortest sequence of: A) any character preceded by a backslash, or B) any thus far unmatched character, until 3. Matching the first unmatched quote. The 2.A case should match before laziness kicks in. The only case when it shouldn’t is if the backslash has already been consumed by a previous escape.