3 ms·

I wonder if, knowing everything we collectively do now, if we had the chance to re-invent regular expressions would they look any different? Given the use case

by mikece 7y ago

I wonder if, knowing everything we collectively do now, if we had the chance to re-invent regular expressions would they look any different? Given the use case I think the complex syntax is inescapable but I can't help but thinking this could be implemented a bit simpler without losing any power.

Boxxed 7y ago
I think syntax is the big whiff in regular expressions. I bet it would feel much more natural to represent them not as a string but as a tree (which they kind of are). Maybe with s-expressions? Would probably be more verbose but I imagine way more readable.
- mikece 7y ago
  S-expressions which make me think of Lisp and functional programming... which I mention because regular expressions as means for defining a pattern against which to find matches has a functional programming feel to it. Perhaps something like Linq in .NET or Map() in JavaScript would work as you're more or less converting the text in a string/document to an array and then running a "does it match or not" function on it and returning the position when there's a match. So something which a functional programming feel makes sense.
dan-robertson 7y ago
Perl6 (ie raku) did do this. Sort of. They made them look a bit more like bnf grammars (or at least the way those grammars are typically written). They ignored whitespace (and allowed them to be written over multiple lines, with comments) which made them more readable and allowed referencing regexps (stored in variables) directly within a regexp. I think they also made some of the syntax nicer (eg non capturing groups or lookaheads), and added notation for common things (e.g. like X* except everything is separated by commas) and long names for things like character classes.
- tyingq 7y ago
  The /x modifier in Perl 5 allows for comments and ignores whitespace. Things like [[:upper:]] are also in Perl 5.
l_t 7y ago
The biggest point of complexity, IMO, is the representation in which the characters to be matched are mixed in with the syntactical elements in a single string. This tends to cause confusion, makes escaping complex, and is just really different from most other ways of expressing logic in programming. Even advanced regex features like extended mode and named captures, which I love, don't get around this fundamental issue with the representation. It does have advantages though, a big one being terseness, and the ability to be expressed in an implementation-independent way. Another is that for some types of regexes, the expression "sort of looks like" the strings it can match against, which can help intuition. I think the way forward is to consider the existing syntax a "terse mode" and to use regex builder libraries [0] as a "verbose mode." The "terse mode" does have merits, though, and I wouldn't want to get rid of it or anything. [0]: For example https://github.com/VerbalExpressions/JSVerbalExpressions https://github.com/VerbalExpressions/JSVerbalExpressions is a library that lets developers express regexps using a fluent API. This changes things from being syntactic elements of a string to just being traditional function calls.
- kbenson 7y ago
  > For example https://github.com/VerbalExpressions/JSVerbalExpressions https://github.com/VerbalExpressions/JSVerbalExpressions is a library that lets developers express regexps using a fluent API. Except from what I can see, it only supports trivial cases. What's a syntax that supports grouping, alternation, a numerical range of number of matches (e.g. \d{3,10}), and the combining of any number of those those? I suspect it quickly devolves to the case where you either use the extended syntax and comments are really what matters, or the terseness is useful in at least not making it so verbose and meandering that it's hard to comprehend just because the amount of boilerplate to usefully express something dwarfs the functional bits. Regular expressions work best when matching a a character level. The non-terse mode of stuff you are grasping at works best when matching at a symbolic level, and has existing for a very long time. They're called grammars. A popular form is BNF[1] (and variants), which will probably look somewhat familiar from technical specifications or RFCs, if you've looked at any before. If you want something beyond regular expressions and you're willing to use a library for it, just use a library that provides grammar parsing. Or use Raku (fka Perl 6), which has support built in (with hooks for calling code at particular parsing points).[2] Hopefully most languages have a good library for parsing grammars at this point. 1: http://matt.might.net/articles/grammars-bnf-ebnf/ http://matt.might.net/articles/grammars-bnf-ebnf/ 2: https://docs.perl6.org/language/grammar_tutorial https://docs.perl6.org/language/grammar_tutorial
mci 7y ago
SNOBOL4 pattern matching may fit your bill, although it is more powerful than regexps [1]. You can play with SNOBOL4 on x86 [2]. [1] http://www.snobol4.org/docs/burks/tutorial/ch4.htm http://www.snobol4.org/docs/burks/tutorial/ch4.htm [2] https://github.com/spitbol/x64 https://github.com/spitbol/x64

by mikece 7y ago

Boxxed 7y ago

I think syntax is the big whiff in regular expressions. I bet it would feel much more natural to represent them not as a string but as a tree (which they kind of are). Maybe with s-expressions? Would probably be more verbose but I imagine way more readable.

mikece 7y ago
S-expressions which make me think of Lisp and functional programming... which I mention because regular expressions as means for defining a pattern against which to find matches has a functional programming feel to it. Perhaps something like Linq in .NET or Map() in JavaScript would work as you're more or less converting the text in a string/document to an array and then running a "does it match or not" function on it and returning the position when there's a match. So something which a functional programming feel makes sense.

dan-robertson 7y ago

Perl6 (ie raku) did do this. Sort of. They made them look a bit more like bnf grammars (or at least the way those grammars are typically written). They ignored whitespace (and allowed them to be written over multiple lines, with comments) which made them more readable and allowed referencing regexps (stored in variables) directly within a regexp. I think they also made some of the syntax nicer (eg non capturing groups or lookaheads), and added notation for common things (e.g. like X* except everything is separated by commas) and long names for things like character classes.

tyingq 7y ago
The /x modifier in Perl 5 allows for comments and ignores whitespace. Things like [[:upper:]] are also in Perl 5.

l_t 7y ago

The biggest point of complexity, IMO, is the representation in which the characters to be matched are mixed in with the syntactical elements in a single string. This tends to cause confusion, makes escaping complex, and is just really different from most other ways of expressing logic in programming. Even advanced regex features like extended mode and named captures, which I love, don't get around this fundamental issue with the representation. It does have advantages though, a big one being terseness, and the ability to be expressed in an implementation-independent way. Another is that for some types of regexes, the expression "sort of looks like" the strings it can match against, which can help intuition. I think the way forward is to consider the existing syntax a "terse mode" and to use regex builder libraries [0] as a "verbose mode." The "terse mode" does have merits, though, and I wouldn't want to get rid of it or anything. [0]: For example https://github.com/VerbalExpressions/JSVerbalExpressions https://github.com/VerbalExpressions/JSVerbalExpressions is a library that lets developers express regexps using a fluent API. This changes things from being syntactic elements of a string to just being traditional function calls.

kbenson 7y ago
> For example https://github.com/VerbalExpressions/JSVerbalExpressions https://github.com/VerbalExpressions/JSVerbalExpressions is a library that lets developers express regexps using a fluent API. Except from what I can see, it only supports trivial cases. What's a syntax that supports grouping, alternation, a numerical range of number of matches (e.g. \d{3,10}), and the combining of any number of those those? I suspect it quickly devolves to the case where you either use the extended syntax and comments are really what matters, or the terseness is useful in at least not making it so verbose and meandering that it's hard to comprehend just because the amount of boilerplate to usefully express something dwarfs the functional bits. Regular expressions work best when matching a a character level. The non-terse mode of stuff you are grasping at works best when matching at a symbolic level, and has existing for a very long time. They're called grammars. A popular form is BNF[1] (and variants), which will probably look somewhat familiar from technical specifications or RFCs, if you've looked at any before. If you want something beyond regular expressions and you're willing to use a library for it, just use a library that provides grammar parsing. Or use Raku (fka Perl 6), which has support built in (with hooks for calling code at particular parsing points).[2] Hopefully most languages have a good library for parsing grammars at this point. 1: http://matt.might.net/articles/grammars-bnf-ebnf/ http://matt.might.net/articles/grammars-bnf-ebnf/ 2: https://docs.perl6.org/language/grammar_tutorial https://docs.perl6.org/language/grammar_tutorial

mci 7y ago

SNOBOL4 pattern matching may fit your bill, although it is more powerful than regexps [1]. You can play with SNOBOL4 on x86 [2]. [1] http://www.snobol4.org/docs/burks/tutorial/ch4.htm http://www.snobol4.org/docs/burks/tutorial/ch4.htm [2] https://github.com/spitbol/x64 https://github.com/spitbol/x64