3 ms·

How is a chain of string checks less brittle and easier to understand? If they are checking for the same pattern, the intrinsic complexity will be the same, the

How is a chain of string checks less brittle and easier to understand? If they are checking for the same pattern, the intrinsic complexity will be the same, the string checks will just add some additional complexity and risk of bugs.

silisili 2y ago
Edited a bit to explain we're just a typical SaaS application. Regex mostly crops up in validations. Just Google the first result for 'email address regex validation.' (?:[a-z0-9!#$%&'+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'+/=?^_`{|}~-]+)|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])")@(?:(?:[a-z0-9](?:[a-z0-9-][a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-][a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\]) How many programmers do you think understand that perfectly at first glance? I've programmed and used regex for decades and can admit, I don't. Is it even correct? Who knows, unless I waste time deciphering both it and the RFC side by side. I'd much rather have a handful of single checks, preferably commented. As is usually the case, performance is not the primary concern.
- bazoom42 2y ago
  Now rewrite the same to a sequence of string checks and show me the code. For a fair comparison you should remove all comments and whitespace as you have done with the above regex. The problem with the above is not the regex per se, the problem is that the email address grammar is really complex for historical reasons. If you insist on validating email syntactically, you can’t avoid that complexity by rewriting to multiple string checks. The solution is to use a library or just perform a simpler validation (eg check for a ‘@‘), since a full syntactic validation does not provide much value anyway - the address might still be invalid anyway.
  - silisili 2y ago
    The difference is, individual checks can be commented and referenced to a particular rule or even line in an RFC. A regex blob is basically 'this is all the rules, RTFM.' And as you mentioned (especially in the case of email validation), they're usually incorrect.
    - bazoom42 2y ago
      You can add comments to regexes, explaining each part. I believe it is called verbose mode. > And as you mentioned (especially in the case of email validation), they're usually incorrect. My point was that the email address might still be invalid despite being syntactically correct, eg if you miss a letter. This is why I don’t understand the obsession with syntax-level email validation. You still need a confirmation mail. But of course there can be a bug in a regex - just as there can be a bug in imperative string-matching code implementing the same pattern.
      - tmtvl 2y ago
        From `perldoc perlre`: > A single "/x" tells the regular expression parser to ignore most whitespace that is neither backslashed nor within a bracketed character class, nor within the characters of a multi-character metapattern like "(?i: ... )". You can use this to break up your regular expression into more readable parts. Also, the "#" character is treated as a metacharacter introducing a comment that runs up to the pattern's closing delimiter, or to the end of the current line if the pattern extends onto the next line.
- defrost 2y ago
  That alone is hard to document and maintain. Coupled with auto gen state diagrams, the current and correct RFC 5322 spec and case notes it's far more defensable. There are some pretty decent RegEx tools about these days. https://regexper.com/#(%3F%3A%5Ba-z0-9!%23%24%25%26%27*%2B%2F%3D%3F%5E_%60%7B%7C%7D%7E-%5D%2B(%3F%3A%5C.%5Ba-z0-9!%23%24%25%26%27*%2B%2F%3D%3F%5E_%60%7B%7C%7D%7E-%5D%2B)*%7C%22(%3F%3A%5B%5Cx01-%5Cx08%5Cx0b%5Cx0c%5Cx0e-%5Cx1f%5Cx21%5Cx23-%5Cx5b%5Cx5d-%5Cx7f%5D%7C%5C%5C%5B%5Cx01-%5Cx09%5Cx0b%5Cx0c%5Cx0e-%5Cx7f%5D)*%22)%40(%3F%3A(%3F%3A%5Ba-z0-9%5D(%3F%3A%5Ba-z0-9-%5D*%5Ba-z0-9%5D)%3F%5C.)%2B%5Ba-z0-9%5D(%3F%3A%5Ba-z0-9-%5D*%5Ba-z0-9%5D)%3F%7C%5C%5B(%3F%3A(%3F%3A(2(5%5B0-5%5D%7C%5B0-4%5D%5B0-9%5D)%7C1%5B0-9%5D%5B0-9%5D%7C%5B1-9%5D%3F%5B0-9%5D))%5C.)%7B3%7D(%3F%3A(2(5%5B0-5%5D%7C%5B0-4%5D%5B0-9%5D)%7C1%5B0-9%5D%5B0-9%5D%7C%5B1-9%5D%3F%5B0-9%5D)%7C%5Ba-z0-9-%5D*%5Ba-z0-9%5D%3A(%3F%3A%5B%5Cx01-%5Cx08%5Cx0b%5Cx0c%5Cx0e-%5Cx1f%5Cx21-%5Cx5a%5Cx53-%5Cx7f%5D%7C%5C%5C%5B%5Cx01-%5Cx09%5Cx0b%5Cx0c%5Cx0e-%5Cx7f%5D)%2B)%5C%5D https://regexper.com/#(%3F%3A%5Ba-z0-9!%23%24%25%26%27*%2B%2...) ^^ Heh. Markup processing error in HN ?? the final ) wasn't captured in the link creation. See https://stackoverflow.com/questions/201323/how-can-i-validate-an-email-address-using-a-regular-expression https://stackoverflow.com/questions/201323/how-can-i-validat... for a working link to the state diagram generator. Even with a handful of single checks there's still the need to compare those, block by block, to the RFC. Assuming RegEx is to be used (I'm not intimidated by RegEx's but I'm general not a fan, preferring custom parsers for many things that are hard or impossible with a RegEx) this is a better approach: https://regex101.com/r/gJ7pU0/1 https://regex101.com/r/gJ7pU0/1 It's a "live" example that includes a test suite and has a parser that annotates blocks. The RegEx expression uses a DEFINE for sub clauses to improve clarity.
  - bazoom42 2y ago
    > I'm not intimidated by RegEx's but I'm general not a fan, preferring custom parsers for many things that are hard or impossible with a RegEx Good call not to use Regex for things that are impossible to do in Regex! But seriously, a custom parser must have some way to recognize individual tokens. If you distinguish parsing and lexing, what tool do you use for lexing? Regexes have a particular purpose: matching patterns of characters. I haven’t seen anyone suggest how to do that in a simpler and cleaner way.
- nathell 2y ago
  > Regex mostly crops up in validations. I’ve just grepped my codebase for regex matchings, and this is not true. The most common use case is matching a filesystem path or a URL that is known to conform to a schema (e.g. file names prefixed with dates) and extracting parts of the schema from it. > Just Google the first result for 'email address regex validation.' That is an abomination and not a good way to validate emails, because, as you say, it’s super complicated and barely understandable. Draw a finite-state automaton corresponding to this regex to see why. Equivalent code written without regex, implementing the same FSA, would easily be >100 LOC and equally incomprehensible. In practice, it’s better to check whether the string contains an @ and maybe a dot, and that’s it. Sure, you won’t be RFC 5322 compliant, but who cares? Your users are much more likely to make a typo in the domain name anyway than misspell the characters that would render the email invalid. Just send an email and see if it arrives. All of the regexes in said codebase of mine are simple. The longest is 75 characters and a straightforward one to check for UUIDs; you can understand it at a glance: [0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}
- fuzztester 2y ago
  iirc perl from 5.x onwards allowed both whitespace (at the right places) and comments, in regexes. using those could make them a lot more readable. can't remember but you might have to specify a flag for it.
  - librasteve 2y ago
    this is the default in https://raku.org https://raku.org
    - fuzztester 2y ago
      good to know, thank you.

silisili 2y ago

Edited a bit to explain we're just a typical SaaS application. Regex mostly crops up in validations. Just Google the first result for 'email address regex validation.' (?:[a-z0-9!#$%&'+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'+/=?^_`{|}~-]+)|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])")@(?:(?:[a-z0-9](?:[a-z0-9-][a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-][a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\]) How many programmers do you think understand that perfectly at first glance? I've programmed and used regex for decades and can admit, I don't. Is it even correct? Who knows, unless I waste time deciphering both it and the RFC side by side. I'd much rather have a handful of single checks, preferably commented. As is usually the case, performance is not the primary concern.

bazoom42 2y ago
Now rewrite the same to a sequence of string checks and show me the code. For a fair comparison you should remove all comments and whitespace as you have done with the above regex. The problem with the above is not the regex per se, the problem is that the email address grammar is really complex for historical reasons. If you insist on validating email syntactically, you can’t avoid that complexity by rewriting to multiple string checks. The solution is to use a library or just perform a simpler validation (eg check for a ‘@‘), since a full syntactic validation does not provide much value anyway - the address might still be invalid anyway.
- silisili 2y ago
  The difference is, individual checks can be commented and referenced to a particular rule or even line in an RFC. A regex blob is basically 'this is all the rules, RTFM.' And as you mentioned (especially in the case of email validation), they're usually incorrect.
  - bazoom42 2y ago
    You can add comments to regexes, explaining each part. I believe it is called verbose mode. > And as you mentioned (especially in the case of email validation), they're usually incorrect. My point was that the email address might still be invalid despite being syntactically correct, eg if you miss a letter. This is why I don’t understand the obsession with syntax-level email validation. You still need a confirmation mail. But of course there can be a bug in a regex - just as there can be a bug in imperative string-matching code implementing the same pattern.
    - tmtvl 2y ago
      From `perldoc perlre`: > A single "/x" tells the regular expression parser to ignore most whitespace that is neither backslashed nor within a bracketed character class, nor within the characters of a multi-character metapattern like "(?i: ... )". You can use this to break up your regular expression into more readable parts. Also, the "#" character is treated as a metacharacter introducing a comment that runs up to the pattern's closing delimiter, or to the end of the current line if the pattern extends onto the next line.
defrost 2y ago
That alone is hard to document and maintain. Coupled with auto gen state diagrams, the current and correct RFC 5322 spec and case notes it's far more defensable. There are some pretty decent RegEx tools about these days. https://regexper.com/#(%3F%3A%5Ba-z0-9!%23%24%25%26%27*%2B%2F%3D%3F%5E_%60%7B%7C%7D%7E-%5D%2B(%3F%3A%5C.%5Ba-z0-9!%23%24%25%26%27*%2B%2F%3D%3F%5E_%60%7B%7C%7D%7E-%5D%2B)*%7C%22(%3F%3A%5B%5Cx01-%5Cx08%5Cx0b%5Cx0c%5Cx0e-%5Cx1f%5Cx21%5Cx23-%5Cx5b%5Cx5d-%5Cx7f%5D%7C%5C%5C%5B%5Cx01-%5Cx09%5Cx0b%5Cx0c%5Cx0e-%5Cx7f%5D)*%22)%40(%3F%3A(%3F%3A%5Ba-z0-9%5D(%3F%3A%5Ba-z0-9-%5D*%5Ba-z0-9%5D)%3F%5C.)%2B%5Ba-z0-9%5D(%3F%3A%5Ba-z0-9-%5D*%5Ba-z0-9%5D)%3F%7C%5C%5B(%3F%3A(%3F%3A(2(5%5B0-5%5D%7C%5B0-4%5D%5B0-9%5D)%7C1%5B0-9%5D%5B0-9%5D%7C%5B1-9%5D%3F%5B0-9%5D))%5C.)%7B3%7D(%3F%3A(2(5%5B0-5%5D%7C%5B0-4%5D%5B0-9%5D)%7C1%5B0-9%5D%5B0-9%5D%7C%5B1-9%5D%3F%5B0-9%5D)%7C%5Ba-z0-9-%5D*%5Ba-z0-9%5D%3A(%3F%3A%5B%5Cx01-%5Cx08%5Cx0b%5Cx0c%5Cx0e-%5Cx1f%5Cx21-%5Cx5a%5Cx53-%5Cx7f%5D%7C%5C%5C%5B%5Cx01-%5Cx09%5Cx0b%5Cx0c%5Cx0e-%5Cx7f%5D)%2B)%5C%5D https://regexper.com/#(%3F%3A%5Ba-z0-9!%23%24%25%26%27*%2B%2...) ^^ Heh. Markup processing error in HN ?? the final ) wasn't captured in the link creation. See https://stackoverflow.com/questions/201323/how-can-i-validate-an-email-address-using-a-regular-expression https://stackoverflow.com/questions/201323/how-can-i-validat... for a working link to the state diagram generator. Even with a handful of single checks there's still the need to compare those, block by block, to the RFC. Assuming RegEx is to be used (I'm not intimidated by RegEx's but I'm general not a fan, preferring custom parsers for many things that are hard or impossible with a RegEx) this is a better approach: https://regex101.com/r/gJ7pU0/1 https://regex101.com/r/gJ7pU0/1 It's a "live" example that includes a test suite and has a parser that annotates blocks. The RegEx expression uses a DEFINE for sub clauses to improve clarity.
- bazoom42 2y ago
  > I'm not intimidated by RegEx's but I'm general not a fan, preferring custom parsers for many things that are hard or impossible with a RegEx Good call not to use Regex for things that are impossible to do in Regex! But seriously, a custom parser must have some way to recognize individual tokens. If you distinguish parsing and lexing, what tool do you use for lexing? Regexes have a particular purpose: matching patterns of characters. I haven’t seen anyone suggest how to do that in a simpler and cleaner way.
nathell 2y ago
> Regex mostly crops up in validations. I’ve just grepped my codebase for regex matchings, and this is not true. The most common use case is matching a filesystem path or a URL that is known to conform to a schema (e.g. file names prefixed with dates) and extracting parts of the schema from it. > Just Google the first result for 'email address regex validation.' That is an abomination and not a good way to validate emails, because, as you say, it’s super complicated and barely understandable. Draw a finite-state automaton corresponding to this regex to see why. Equivalent code written without regex, implementing the same FSA, would easily be >100 LOC and equally incomprehensible. In practice, it’s better to check whether the string contains an @ and maybe a dot, and that’s it. Sure, you won’t be RFC 5322 compliant, but who cares? Your users are much more likely to make a typo in the domain name anyway than misspell the characters that would render the email invalid. Just send an email and see if it arrives. All of the regexes in said codebase of mine are simple. The longest is 75 characters and a straightforward one to check for UUIDs; you can understand it at a glance: [0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}
fuzztester 2y ago
iirc perl from 5.x onwards allowed both whitespace (at the right places) and comments, in regexes. using those could make them a lot more readable. can't remember but you might have to specify a flag for it.
- librasteve 2y ago
  this is the default in https://raku.org https://raku.org
  - fuzztester 2y ago
    good to know, thank you.