3 ms·

Parsing strings assembled out of strings is classically bug-prone. In the alternative I'm pointing out, you don't fill the holes with other strings, you fill th

Parsing strings assembled out of strings is classically bug-prone. In the alternative I'm pointing out, you don't fill the holes with other strings, you fill them with already-parsed regex objects. I think it's a shame Python didn't follow this design, which predated f-strings.

pdonis 4y ago
Can you give an example of a regex engine that has the design you describe?
- abecedarius 4y ago
  I must have a serious bug in my writing about this (sorry), because this was never about regex engines -- it's about literals and domain-specific sublanguages in general. Composing DSL programs by string concatenation is such a famous source of security bugs you see it in top-10 lists. I linked to the very similar example of a PEG-parsing DSL. But any regex engine that can work with a parse tree shows the same principle, e.g. https://edicl.github.io/cl-ppcre/#create-scanner2 https://edicl.github.io/cl-ppcre/#create-scanner2
  - pdonis 4y ago
    > this was never about regex engines Since you explicitly talked about filling the fields in an f string with already parsed regex objects instead of strings, it's hard to see what else you could mean. But even if I s/regex engine/DSL parsing engine in general/, I would like to see an actual example of a language or library where I can have a string like a Python f-string whose fields can be filled with some kind of parsed "engine" object instead of another string. > Composing DSL programs by string concatenation is such a famous source of security bugs you see it in top-10 lists. I don't see how composing DSL programs by filling in string fields with parsed "engine" objects is much better. I personally don't like regexes in general because I find them too hard to reason about unless they're extremely simple (and regexes that simple usually aren't necessary). I would rather try to write library functions (which might include functions that build other functions) in the same language as the rest of my program.
    - abecedarius 4y ago
      Yeah, sorry I didn't explain tagged template literals, just linked to an example. You could make one called, say, rx for regex. Then rx`${a}|${b}` would evaluate to exactly the same result as a tree constructor call like regex_or(a, b) given corresponding definitions of rx and of regex_or. There's never any question of whether a and b are escaped right. So it brings the composability advantage of the library functions you prefer, to people who want to write these concrete-syntax regular expressions that started the whole thread.
joshuamorton 4y ago
> I think it's a shame Python didn't follow this design I mean python's regex module was added in 1997 and hasn't fundamentally changed since. I don't think that concept was super common in '97.
- abecedarius 4y ago
  I meant the design of f-strings, not of regexes. I must've written my comment very badly since almost all of these replies seem to take this misinterpretation. (E did have this concept back in '97, iirc, though yeah I wasn't expecting Guido to have run into it then, that wasn't when I meant.)
  - joshuamorton 4y ago
    I think the issue is that f-strings have nothing specifically to do with regex specifically, or parsing generally. They're a formatting API. So the complaint that you wish the formatting API made regexes a first class citizen is odd and suggests a misunderstanding somewhere.
    - abecedarius 4y ago
      Evidently I was still unclear: An f-string fills holes in a format string to build a string. A template literal parses a template with holes to produce any datatype you like, filling it with arguments of any appropriate type. This does for Javascript what f-strings do for Python (with similar syntax and simplicity), but also more. Besides the greater expressiveness, it can catch bugs, in the same way that Lisp macros are safer than C macros. It's not hardwiring regexes, it's not hardwiring any datatype: it calls a function that you name in the tag, which you can define to do whatever parsing and filling is proper. Lisp's quasiquotation is similar in spirit though different in appearance. This is my last try to explain here. I guess this thread shows that tagged template strings are much less well known on HN than I thought. (Yes it also shows me I was unusually bad at communicating.) > the complaint that you wish the formatting API made regexes a first class citizen is odd That was not what I was trying to say. The template literal mechanism knows nothing about regexes. Regexes are just one particular type and one particular syntax.
thaumasiotes 4y ago
> Parsing strings assembled out of strings is classically bug-prone. I think the bigger problem is not so much that regexes are generally defined by strings specifying the desired regex, but that at some point between the development of printf and the development of regex libraries people forgot that they could use whatever escape character they wanted when implementing a new conceptual data type. In C, the compiler deals with strings, and the printf function deals with strings, and they try not to conflict with each other by assigning the escape character \ to the compiler while printf uses % instead. But in Java, and Python, and presumably many, many other languages, some idiot decided that if strings used \ for their escape character, regex functions should also use \ for their escape character. Since they accept strings, suddenly the regex escape character is actually "\\". How do you match a single literal backslash? "\\\\", obviously. What's wrong with that?

pdonis 4y ago

Can you give an example of a regex engine that has the design you describe?

abecedarius 4y ago
I must have a serious bug in my writing about this (sorry), because this was never about regex engines -- it's about literals and domain-specific sublanguages in general. Composing DSL programs by string concatenation is such a famous source of security bugs you see it in top-10 lists. I linked to the very similar example of a PEG-parsing DSL. But any regex engine that can work with a parse tree shows the same principle, e.g. https://edicl.github.io/cl-ppcre/#create-scanner2 https://edicl.github.io/cl-ppcre/#create-scanner2
- pdonis 4y ago
  > this was never about regex engines Since you explicitly talked about filling the fields in an f string with already parsed regex objects instead of strings, it's hard to see what else you could mean. But even if I s/regex engine/DSL parsing engine in general/, I would like to see an actual example of a language or library where I can have a string like a Python f-string whose fields can be filled with some kind of parsed "engine" object instead of another string. > Composing DSL programs by string concatenation is such a famous source of security bugs you see it in top-10 lists. I don't see how composing DSL programs by filling in string fields with parsed "engine" objects is much better. I personally don't like regexes in general because I find them too hard to reason about unless they're extremely simple (and regexes that simple usually aren't necessary). I would rather try to write library functions (which might include functions that build other functions) in the same language as the rest of my program.
  - abecedarius 4y ago
    Yeah, sorry I didn't explain tagged template literals, just linked to an example. You could make one called, say, rx for regex. Then rx`${a}|${b}` would evaluate to exactly the same result as a tree constructor call like regex_or(a, b) given corresponding definitions of rx and of regex_or. There's never any question of whether a and b are escaped right. So it brings the composability advantage of the library functions you prefer, to people who want to write these concrete-syntax regular expressions that started the whole thread.

joshuamorton 4y ago

> I think it's a shame Python didn't follow this design I mean python's regex module was added in 1997 and hasn't fundamentally changed since. I don't think that concept was super common in '97.

abecedarius 4y ago
I meant the design of f-strings, not of regexes. I must've written my comment very badly since almost all of these replies seem to take this misinterpretation. (E did have this concept back in '97, iirc, though yeah I wasn't expecting Guido to have run into it then, that wasn't when I meant.)
- joshuamorton 4y ago
  I think the issue is that f-strings have nothing specifically to do with regex specifically, or parsing generally. They're a formatting API. So the complaint that you wish the formatting API made regexes a first class citizen is odd and suggests a misunderstanding somewhere.
  - abecedarius 4y ago
    Evidently I was still unclear: An f-string fills holes in a format string to build a string. A template literal parses a template with holes to produce any datatype you like, filling it with arguments of any appropriate type. This does for Javascript what f-strings do for Python (with similar syntax and simplicity), but also more. Besides the greater expressiveness, it can catch bugs, in the same way that Lisp macros are safer than C macros. It's not hardwiring regexes, it's not hardwiring any datatype: it calls a function that you name in the tag, which you can define to do whatever parsing and filling is proper. Lisp's quasiquotation is similar in spirit though different in appearance. This is my last try to explain here. I guess this thread shows that tagged template strings are much less well known on HN than I thought. (Yes it also shows me I was unusually bad at communicating.) > the complaint that you wish the formatting API made regexes a first class citizen is odd That was not what I was trying to say. The template literal mechanism knows nothing about regexes. Regexes are just one particular type and one particular syntax.

thaumasiotes 4y ago

> Parsing strings assembled out of strings is classically bug-prone. I think the bigger problem is not so much that regexes are generally defined by strings specifying the desired regex, but that at some point between the development of printf and the development of regex libraries people forgot that they could use whatever escape character they wanted when implementing a new conceptual data type. In C, the compiler deals with strings, and the printf function deals with strings, and they try not to conflict with each other by assigning the escape character \ to the compiler while printf uses % instead. But in Java, and Python, and presumably many, many other languages, some idiot decided that if strings used \ for their escape character, regex functions should also use \ for their escape character. Since they accept strings, suddenly the regex escape character is actually "\\". How do you match a single literal backslash? "\\\\", obviously. What's wrong with that?