12 ms·

Why are regular expressions difficult?

12 points by nojito 4y ago

PaulHoule 4y ago
I don't believe regular expressions are difficult, at least compared to structures that are a step up in expressiveness such as parser generators. The ergonomic problems of parser generators are worth quite a bit of analysis, but the basic difference is that people can learn how to use regular expressions without understanding how regexes are implemented but practically you have to know how parser generators work to use a parser generator. Anybody who can write a parser generator can easily work around the things that are difficult with parser generators.
- fargle 4y ago
  totally agree. the problem isn't "regex's are hard" - it's "math is hard". Regex's (especially real regexes that are actually regular languages) are intuitive and terse, but direct and efficient. What's hard is that even regular grammars, let alone CFG, let along non-CFG, are hard for people to grasp. The problem is HARD, not the solution!
egberts1 4y ago
To write regex with relative ease does require mastering some basic fundamentals of Regex. For example to have a regex that can remove // from inline comment of an INI-format text file. (((\x27[ \!\"\#\$\%\&\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\x27\s*)|(\"[ \!\#\$\%\&\x27\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\"\s*)|(\/([ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+[ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*)|([ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*))*)*)([;#]+)*.*$ That regex works very well and it’s breakdown can be better explained using various online Regex testers Off-topically, I’ve got another next-level problem, and that is how to convert Regex for use within a bash shell script. It’s nasty. However, I have listed several online Regex testers so you can better understand the result of (my) Regex creation, one regex character at a time. https://www.debuggex.com/ https://www.debuggex.com/ https://extendsclass.com/regex/4d49f33 https://extendsclass.com/regex/4d49f33 https://www.freeformatter.com/regex-tester.html https://www.freeformatter.com/regex-tester.html https://regexr.com/6gv39 https://regexr.com/6gv39 More online testers that I couldn’t get to work with are detailed in https://github.com/egberts/bash-ini-file/issues/1 https://github.com/egberts/bash-ini-file/issues/1
- Veserv 4y ago
  A regular expression like that is hard to decipher because it is a giant pile of unformatted duplication. It is challenging to read in the same way that a program written by manually inlining every function call in the entire program and then putting everything onto the same line (because you have a non-newline statement separator) is hard to read. Let us see how to use the advanced technology of factoring out common expressions into "subroutines" makes it much easier to understand. Note, I have no idea of what the regex you are doing actually does, so I am doing things purely mechanically, and I have only basic working-memory knowledge of regexes. First, let us restructure it: ( ( ( \x27 [ \!\"\#\$\%\&\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]* \x27\s* ) |( \" [ \!\#\$\%\&\x27\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]* \"\s* ) |( \/( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+ [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) |( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) )* )* ) ([;#]+)*.*$ Now let us reorganize the clearly common expressions: ( ( ( \x27 [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\"]* \x27\s* ) |( \" [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\x27]* \"\s* ) |( \/( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+ [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) |( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) )* )* ) ([;#]+)*.*$ Now let us factor out the common expressions: common = { \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~} separator = {\;\#\/\\} ( ( ( \x27 [common + separator + {\"}]* \x27\s* ) |( \" [common + separator + {\x27}]* \"\s* ) |( \/( [common]+[common]* ) |( [common]* ) )* )* ) ([;#]+)*.*$ At this point, going any further likely requires a deeper understanding of what it is actually trying to accomplish, but the purely mechanical transformations have dramatically reduced complexity of the regex to something in the realm of comprehensibility.
Planktonne 4y ago
Regular expressions aren't difficult. People think regular expressions are difficult because they're taught either not at all, or very badly. If you learn them deliberately, moving through the unfolding complexity of patterns, they're actually both beautiful and intuitive. I think most programmers first come to regular expressions when they encounter a problem that can't be efficiently solved in any other way. The sort of problem that can only be solved by regex is also the sort of problem which is solved by complex regex. This means that the first introduction is to be thrown right in at the very, very deepest end. Regex becomes a confusing and arcane thing, and the programmer resolves to rarely use them. However, it doesn't have to be like that. If you teach regex before they're 100% required - when they're used to solve problems that are biddable and have multiple possible solutions - you can start to get a feel for the syntax. That makes regex feel accessible, and soon you can start to work with even the complex ones.
deleted 4y ago
[deleted]
silent_cal 4y ago
Regex is completely unreadable to the uninitiated. Why not try to develop a more readable syntax like SQL has?
hackarama 4y ago
Regex isn't difficult?!?! Maybe I don't have the same big-brains that you folks do, but the more I study these, the more confused I seem to wind up becoming.
tamaker 4y ago
thought it was just me. good read.

12 points by nojito 4y ago

I don't believe regular expressions are difficult, at least compared to structures that are a step up in expressiveness such as parser generators. The ergonomic problems of parser generators are worth quite a bit of analysis, but the basic difference is that people can learn how to use regular expressions without understanding how regexes are implemented but practically you have to know how parser generators work to use a parser generator. Anybody who can write a parser generator can easily work around the things that are difficult with parser generators.

fargle 4y ago
totally agree. the problem isn't "regex's are hard" - it's "math is hard". Regex's (especially real regexes that are actually regular languages) are intuitive and terse, but direct and efficient. What's hard is that even regular grammars, let alone CFG, let along non-CFG, are hard for people to grasp. The problem is HARD, not the solution!

egberts1 4y ago

To write regex with relative ease does require mastering some basic fundamentals of Regex. For example to have a regex that can remove // from inline comment of an INI-format text file. (((\x27[ \!\"\#\$\%\&\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\x27\s*)|(\"[ \!\#\$\%\&\x27\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\"\s*)|(\/([ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+[ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*)|([ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*))*)*)([;#]+)*.*$ That regex works very well and it’s breakdown can be better explained using various online Regex testers Off-topically, I’ve got another next-level problem, and that is how to convert Regex for use within a bash shell script. It’s nasty. However, I have listed several online Regex testers so you can better understand the result of (my) Regex creation, one regex character at a time. https://www.debuggex.com/ https://www.debuggex.com/ https://extendsclass.com/regex/4d49f33 https://extendsclass.com/regex/4d49f33 https://www.freeformatter.com/regex-tester.html https://www.freeformatter.com/regex-tester.html https://regexr.com/6gv39 https://regexr.com/6gv39 More online testers that I couldn’t get to work with are detailed in https://github.com/egberts/bash-ini-file/issues/1 https://github.com/egberts/bash-ini-file/issues/1

Veserv 4y ago
A regular expression like that is hard to decipher because it is a giant pile of unformatted duplication. It is challenging to read in the same way that a program written by manually inlining every function call in the entire program and then putting everything onto the same line (because you have a non-newline statement separator) is hard to read. Let us see how to use the advanced technology of factoring out common expressions into "subroutines" makes it much easier to understand. Note, I have no idea of what the regex you are doing actually does, so I am doing things purely mechanically, and I have only basic working-memory knowledge of regexes. First, let us restructure it: ( ( ( \x27 [ \!\"\#\$\%\&\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]* \x27\s* ) |( \" [ \!\#\$\%\&\x27\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]* \"\s* ) |( \/( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+ [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) |( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) )* )* ) ([;#]+)*.*$ Now let us reorganize the clearly common expressions: ( ( ( \x27 [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\"]* \x27\s* ) |( \" [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\x27]* \"\s* ) |( \/( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+ [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) |( [ \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]* ) )* )* ) ([;#]+)*.*$ Now let us factor out the common expressions: common = { \!\$\%\&\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~} separator = {\;\#\/\\} ( ( ( \x27 [common + separator + {\"}]* \x27\s* ) |( \" [common + separator + {\x27}]* \"\s* ) |( \/( [common]+[common]* ) |( [common]* ) )* )* ) ([;#]+)*.*$ At this point, going any further likely requires a deeper understanding of what it is actually trying to accomplish, but the purely mechanical transformations have dramatically reduced complexity of the regex to something in the realm of comprehensibility.

Planktonne 4y ago

Regular expressions aren't difficult. People think regular expressions are difficult because they're taught either not at all, or very badly. If you learn them deliberately, moving through the unfolding complexity of patterns, they're actually both beautiful and intuitive. I think most programmers first come to regular expressions when they encounter a problem that can't be efficiently solved in any other way. The sort of problem that can only be solved by regex is also the sort of problem which is solved by complex regex. This means that the first introduction is to be thrown right in at the very, very deepest end. Regex becomes a confusing and arcane thing, and the programmer resolves to rarely use them. However, it doesn't have to be like that. If you teach regex before they're 100% required - when they're used to solve problems that are biddable and have multiple possible solutions - you can start to get a feel for the syntax. That makes regex feel accessible, and soon you can start to work with even the complex ones.

deleted 4y ago

[deleted]

silent_cal 4y ago

Regex is completely unreadable to the uninitiated. Why not try to develop a more readable syntax like SQL has?

hackarama 4y ago

Regex isn't difficult?!?! Maybe I don't have the same big-brains that you folks do, but the more I study these, the more confused I seem to wind up becoming.

tamaker 4y ago

thought it was just me. good read.