4 ms·

Sometimes I feel like the difference between a junior, intermediate, and senior developers is that the junior hasn't yet figured out how to use regexps, the int

by Lazare 11y ago

Sometimes I feel like the difference between a junior, intermediate, and senior developers is that the junior hasn't yet figured out how to use regexps, the intermediate dev has, and the senior dev has figured out not to. I kid, but...really. The number of times I've seen people burnt by non-trivial regexps in production code is absurd. "Oh, I need to mangle this CSV that's in the wrong format"? Sure, write a one-off regexp. "Hm, there might be a security vulnerability in this URL parameter, I need some way to filter out bad inputs..." Oh hell no. Literally two weeks ago I tracked down a crazy bug to a regexp someone had written to try and correct mistyped email addresses in a signup form (?!) that deleted "invalid" characters. Like a "+". So many facepalms for one short line of code...

anotheryou 11y ago
out of curiosity: what would you use for the URL validation? (but i get this is difficult: https://gist.github.com/dperini/729294 https://gist.github.com/dperini/729294 )
- anton_gogolev 11y ago
  I prefer straight up imperative code (or, better yet, whatever Url/Uri API that is provided by the platform).
- Lazare 11y ago
  The general rule is that any tricky security code should have as many eyeballs on it as possible, so I'd probably start by seeing if my framework had solid support for this, or failing that, see what popular libraries for my platform exist. In this particular case, the Google Caja project[1] is a good starting place for most HTML/JS/CSS sanitization needs (although the project has a much larger scope than just that); and I think the 'sanitizer' package on npm is a fairly popular wrapper/port of it's basic sanitizing code, and I believe ruby/php/python have their own but I couldn't name them offhand. But it would depend on the exact attack vector you're trying to stop, eg, XSS, remote shell via filename params, etc. If I had to write it myself, I'd probably go for something as braindead as possible; probably a bunch of nested loops backed by some thorough tests. Regexps are great for magic one liners, but magic one liners are antithetical to good security. [1]: https://developers.google.com/caja/ https://developers.google.com/caja/
- nly 11y ago
  Well, it's clear to see why people are lead astray. In this case, RFC 3986, which defines what a URI actually is, itself actually proposes a regex[0]. It even claims, and I quote, 'the "first-match-wins" algorithm is identical to the "greedy" disambiguation method used by POSIX regular expressions'. Of course, it doesn't actually validate anything... it just performs rudimentary field extraction. Trying to do anything beyond that with regex is madness. The ABNF grammar is fairly simple though and, I think, free of ambiguities. I've had luck converting it straight in to PEG form. It's not a trivial or wholly useful endeavour though, and, if you do so, remember to check the errata. For HTTP you'll also have to add in the changes from RFC 7230[1]. Oh, and of course, none of this validates DNS names, their labels, etc. for length, the "LDH rule", or the public suffix list[2], or IP addresses to check whether they have publicly routable prefixes. Bottom line is, if you want to validate a URL, the best thing to do, much like e-mail, is to just try and GET it. [0] https://tools.ietf.org/html/rfc3986#appendix-B https://tools.ietf.org/html/rfc3986#appendix-B [1] https://tools.ietf.org/html/rfc7230#section-2.7.1 https://tools.ietf.org/html/rfc7230#section-2.7.1 [2] https://publicsuffix.org/ https://publicsuffix.org/
mbrock 11y ago
The famous "Gruber URL regexp", before an update in 2014, would run in exponential time given some URLs that contained parens. I noticed this when a web app I worked on froze on certain pages. Runaway regexp matching is one of the easiest ways to really lock a JavaScript thread.
- johnmaguire2013 11y ago
  I was also hit by this one in my pet IRC bot. :) URL matching would fail (occasionally) and the whole bot would freeze up. :<
jerf 11y ago
The rule I've developed for myself is never clean up data for security purposes. There are some cases where it is valid to examine the data for validity; for instance, there's many places where you'd like to filter null characters or "all of the ASCII control characters except the whitespace", and it's valid to look for those things. But if you find them, reject the input. In theory you'd think you can clean it up, but in practice it turns out to be just too dangerous. Too many concrete instances of real-world failure, and reality always trumps theory.
deleted 11y ago
[deleted]
BinaryIdiot 11y ago
I completely agree. Honestly regular expressions are hard to get a handle on anyway. I've done software development for about 12 years now and I probably run into a regular expression pretty regularly and I still find them hard to read and understand. Almost every single time they do things that can easily be accomplished with a loop and a tiny bit of string manipulation and, honestly, even if the regex was a little faster it's unlikely to be so much faster than you need to make the micro-optimization of using regex instead of some looping. I try to avoid regex where possible and in the event that I do need to use it I try to document it the best I can and to make sure it's as simple as possible to avoid weird issues like this.

by Lazare 11y ago

anotheryou 11y ago

out of curiosity: what would you use for the URL validation? (but i get this is difficult: https://gist.github.com/dperini/729294 https://gist.github.com/dperini/729294 )

anton_gogolev 11y ago
I prefer straight up imperative code (or, better yet, whatever Url/Uri API that is provided by the platform).
Lazare 11y ago
The general rule is that any tricky security code should have as many eyeballs on it as possible, so I'd probably start by seeing if my framework had solid support for this, or failing that, see what popular libraries for my platform exist. In this particular case, the Google Caja project[1] is a good starting place for most HTML/JS/CSS sanitization needs (although the project has a much larger scope than just that); and I think the 'sanitizer' package on npm is a fairly popular wrapper/port of it's basic sanitizing code, and I believe ruby/php/python have their own but I couldn't name them offhand. But it would depend on the exact attack vector you're trying to stop, eg, XSS, remote shell via filename params, etc. If I had to write it myself, I'd probably go for something as braindead as possible; probably a bunch of nested loops backed by some thorough tests. Regexps are great for magic one liners, but magic one liners are antithetical to good security. [1]: https://developers.google.com/caja/ https://developers.google.com/caja/
nly 11y ago
Well, it's clear to see why people are lead astray. In this case, RFC 3986, which defines what a URI actually is, itself actually proposes a regex[0]. It even claims, and I quote, 'the "first-match-wins" algorithm is identical to the "greedy" disambiguation method used by POSIX regular expressions'. Of course, it doesn't actually validate anything... it just performs rudimentary field extraction. Trying to do anything beyond that with regex is madness. The ABNF grammar is fairly simple though and, I think, free of ambiguities. I've had luck converting it straight in to PEG form. It's not a trivial or wholly useful endeavour though, and, if you do so, remember to check the errata. For HTTP you'll also have to add in the changes from RFC 7230[1]. Oh, and of course, none of this validates DNS names, their labels, etc. for length, the "LDH rule", or the public suffix list[2], or IP addresses to check whether they have publicly routable prefixes. Bottom line is, if you want to validate a URL, the best thing to do, much like e-mail, is to just try and GET it. [0] https://tools.ietf.org/html/rfc3986#appendix-B https://tools.ietf.org/html/rfc3986#appendix-B [1] https://tools.ietf.org/html/rfc7230#section-2.7.1 https://tools.ietf.org/html/rfc7230#section-2.7.1 [2] https://publicsuffix.org/ https://publicsuffix.org/

mbrock 11y ago

The famous "Gruber URL regexp", before an update in 2014, would run in exponential time given some URLs that contained parens. I noticed this when a web app I worked on froze on certain pages. Runaway regexp matching is one of the easiest ways to really lock a JavaScript thread.

johnmaguire2013 11y ago
I was also hit by this one in my pet IRC bot. :) URL matching would fail (occasionally) and the whole bot would freeze up. :<

jerf 11y ago

The rule I've developed for myself is never clean up data for security purposes. There are some cases where it is valid to examine the data for validity; for instance, there's many places where you'd like to filter null characters or "all of the ASCII control characters except the whitespace", and it's valid to look for those things. But if you find them, reject the input. In theory you'd think you can clean it up, but in practice it turns out to be just too dangerous. Too many concrete instances of real-world failure, and reality always trumps theory.

deleted 11y ago

[deleted]

BinaryIdiot 11y ago

I completely agree. Honestly regular expressions are hard to get a handle on anyway. I've done software development for about 12 years now and I probably run into a regular expression pretty regularly and I still find them hard to read and understand. Almost every single time they do things that can easily be accomplished with a loop and a tiny bit of string manipulation and, honestly, even if the regex was a little faster it's unlikely to be so much faster than you need to make the micro-optimization of using regex instead of some looping. I try to avoid regex where possible and in the event that I do need to use it I try to document it the best I can and to make sure it's as simple as possible to avoid weird issues like this.