4 ms·

5 Regular Expressions Every Web Programmer Should Know

57 points by soundsop 18y ago

silentbicycle 18y ago
Instead of learning $bullet_point_count regular expressions by rote, just learn how they work and make your own as you need them. They really aren't hard to use, they're just incredibly compact, which makes them look unintelligible at first. _Mastering Regular Expressions_ is the obvious classic, but a decent intro to Perl or Python will probably get to them eventually. (If you use Emacs, it's particularly easy to learn them by experimenting with M-x re-builder.) Also: Not everything uses the same RE implementation, some use non-standard extensions. Don't become too drunk with power yet, though! There's a lot they're just not capable of doing (most notably, balancing tags around arbitrarily nested expressions). Once you have REs down, learning a lexer/parser is the next step. :)
- tdavis 18y ago
  Exactly! Learn how to write regular expressions because they are super useful, then immediately disregard the boiler-plate ones provided in this article because any self-respecting language's stdlib (or extremely common libraries/frameworks) will handle all these specific cases in an abstracted way. Not to mention, learning only these doesn't mean you understand regex and not all the features of most regex engines are covered here (lookaheads, lookbacks, etc. and so on)
- gruseom 18y ago
  If you use Emacs, it's particularly easy to learn them by experimenting with M-x re-builder. Hey! I did not know about this, and it looks really useful. I've used http://www.newartisans.com/blog_files/regex.tool.for.emacs.php http://www.newartisans.com/blog_files/regex.tool.for.emacs.p... and it's good, but not as well integrated. I wish I knew a better way to get a survey of what's available in the Emacs world - something that would have clued me in to this command, for example... any tips? Edit: I want to comment on this as well: learning a lexer/parser is the next step Recently I've had occasion to write a few parsers. In the past, I'd always used parser generators (yacc and antlr) on the assumption that they made things easier. For this project that wasn't an option, so I bit the bullet and did the recursive descent thing. To my surprise, it turned out to be way, way easier than I expected. Moreover, in at least one case (a complex one), the hand-written parser code turned out shorter than the (cl-) yacc version, as well as handling more cases and reporting errors better. The morals of the story are: (1) Don't assume something is hard without trying it; (2) It's not hard to write a classical recursive-descent parser and the skill is a lot more useful than you may realize. Wish I'd learned that a long time ago.
  - litewulf 18y ago
    That being said, RDPs have quite a bit of boilerplate that may obscure your grammar. (And small bugs in a RDP are sometimes very hard to catch...)
  - mark_h 18y ago
    I wish I knew a better way to get a survey of what's available in the Emacs world - something that would have clued me in to this command, for example... any tips? http://planet.emacsen.org/ http://planet.emacsen.org/ often has useful stuff. The emacs wiki (http://emacswiki.org/ http://emacswiki.org/) is often where you end up after googling and has a heap of information (although even it isn't complete!). You could try http://www.emacswiki.org/cgi-bin/wiki/RandomPage http://www.emacswiki.org/cgi-bin/wiki/RandomPage when you're bored :)
  - silentbicycle 18y ago
    I second Emacswiki as suggested, but I find quite a bit by searching through the Emacs and Elisp info pages and using M-x apropos / M-x apropos-variable. It doesn't always work though - to find something, you generally need to know what Emacs calls it.
- Darmani 18y ago
  If I understand what you mean, you actually can balance tags around arbitrarily nested expressions if your regex parser supports recursion. The Ruby Way has an example of using recursive regexes to balance parentheses.
  - silentbicycle 18y ago
    Once your regexes are getting that complicated, though, you're probably better off specifying things in terms of a BNF-based parser (or a recursive-descent parser, depending on the grammar).
- jimbokun 18y ago
  The article seems like a good tutorial for learning regex constructions, with the examples chosen good motivation for learning.
tocomment 18y ago
meh disappointed. Could any of you guys give me one liners for email and URL validation?
- briansmith 18y ago
  Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/email.html. My URL (IRI) validation regex is a very complicated work in progress and depends on being post-processed by a Python function to handle Unicode characters outside the BMP. It has hundreds of lines of unit tests (not shown): unreserved = ur'(?:[A-Za-z0-9\-\._~]|%s)' % ucschar pct_encoded= ur'%[0-9A-Fa-f]{2}' sub_delims = ur"[!$&'()*+,;=]" pchar = r'(?:%s|%s|%s|[:@])' % (unreserved, pct_encoded, sub_delims) dec_octet = ur'(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])' ipv4address = ur'%s\.%s\.%s\.%s' % (dec_octet, dec_octet, dec_octet, dec_octet) h16 = ur'(?:[0-9A-Fa-f]{1,4})' # 16 bits of address represented in hexadecima ls32 = ur'(?:%s:%s|%s)' % (h16, h16, ipv4address) # least-significant 32 bits of address ipv6address = ( u'(?:' + ur'(?:h16:){6}ls32' + u'|' + ur'::(?:h16:){5}ls32' + u'|' + ur'(h16)?::(?:h16:){4}ls32' + u'|' + ur'((?:h16:){0,1}h16)?::(?:h16:){3}ls32' + u'|' + ur'((?:h16:){0,2}h16)?::(?:h16:){2}ls32' + u'|' + ur'((?:h16:){0,3}h16)?::(?:h16:){1}ls32' + u'|' + ur'((?:h16:){0,4}h16)?::ls32' + u'|' + ur'((?:h16:){0,5}h16)?::h16' + u'|' + ur'((?:h16:){0,6}h16)?::' + u')' ).replace(u"h16", h16).replace(u"ls32",ls32) ipvfuture = ur'v[0-9A-Fa-f]+\.(?:%s|%s|:)+' % (unreserved, sub_delims) ip_literal = ur'\[(?:%s|%s)\]' % (ipv6address, ipvfuture) scheme = ur'(?:[A-Za-z][A-Za-z0-9+\-\.]*)' userinfo = ur'(?:%s|%s|%s|:)*' % (unreserved, pct_encoded, sub_delims) reg_name = ur'(?:%s|%s|%s)*' % (unreserved, pct_encoded, sub_delims) host = ur'(?:%s|%s|%s)' % (ip_literal, ipv4address, reg_name) port = ur'[0-9]*' authority = ur'(?:(?:%s@)?%s(?:\:%s)?)' % (userinfo, host, port) segment = ur'%s*' % pchar segment_nz = ur'%s+' % pchar path_abempty = ur'(?:/%s)*' % ( segment) path_absolute = ur'(?:/(?:%s(?:/%s)*)?)' % (segment_nz, segment) path_rootless = ur'(?:%s(?:/%s)*)' % (segment_nz, segment) path_empty = ur'' hier_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_rootless, path_empty) private = ur'[\ue000-\uF8FF]' # see replaceSurrogatePairs for handling of non-BMP chars query = ur'(?:%s|[/?]|%s)*' % (pchar, private) fragment = ur'(?:%s|[/?])*' % (pchar) iri = ur'%s:%s(?:\?%s)?(?:#%s)?' % (scheme, hier_part, query, fragment) segment_nz_nc = ur'(?:%s|%s|%s|@)+' % (unreserved, pct_encoded, sub_delims) path_noscheme = ur'(?:%s(/%s)*)' % (segment_nz_nc, segment) relative_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_noscheme, path_empty) relative_ref = ur'(?:%s(?:\?%s)?(?:#%s)?)' % (relative_part, query, fragment) iri_reference = ur'(?:%s|%s)' % (iri, relative_ref)
BrandonM 18y ago
The first tip is okay, but the latter 4 are just horrible. Any decent language will have libraries for parsing HTML or e-mails addresses. A regex is sure to come up short and be very fragile. I once had to maintain some screen-scraping code that was written in Python using regular expressions. By the time I inherited it, half of the functionality no longer worked. It would have been much better off using a library like BeautifulSoup, both in terms of development time and maintainability. BeautifulSoup alone takes care of REs 2 and 3, and there are standard libraries in Python that take care of 4 and 5. Why reinvent (less robustly, I might add) the wheel when a simple API already exists in many languages?
Erwin 18y ago
I wouldn't use a single regexp for complete username validation -- if it fails, all you can display is a generic "username is not valid, it must obey rules X, Y & Z" message. I'd check min and max length separately and display an appropriate error message for that. Also ignore leading/trailing spaces; or you'll end up puzzled why you have two "bob@example.org" users in your database even with appropriate database constraint, and bob mails you saying he can't login on the account he just paid for.
antirez 18y ago
5 Regular Expressions Every Web Programmer should now how to write... if he claims I knows regexps. The problem is that in my experience a minimal part of web developers really know regexps.

57 points by soundsop 18y ago

Instead of learning $bullet_point_count regular expressions by rote, just learn how they work and make your own as you need them. They really aren't hard to use, they're just incredibly compact, which makes them look unintelligible at first. _Mastering Regular Expressions_ is the obvious classic, but a decent intro to Perl or Python will probably get to them eventually. (If you use Emacs, it's particularly easy to learn them by experimenting with M-x re-builder.) Also: Not everything uses the same RE implementation, some use non-standard extensions. Don't become too drunk with power yet, though! There's a lot they're just not capable of doing (most notably, balancing tags around arbitrarily nested expressions). Once you have REs down, learning a lexer/parser is the next step. :)

tdavis 18y ago
Exactly! Learn how to write regular expressions because they are super useful, then immediately disregard the boiler-plate ones provided in this article because any self-respecting language's stdlib (or extremely common libraries/frameworks) will handle all these specific cases in an abstracted way. Not to mention, learning only these doesn't mean you understand regex and not all the features of most regex engines are covered here (lookaheads, lookbacks, etc. and so on)
gruseom 18y ago
If you use Emacs, it's particularly easy to learn them by experimenting with M-x re-builder. Hey! I did not know about this, and it looks really useful. I've used http://www.newartisans.com/blog_files/regex.tool.for.emacs.php http://www.newartisans.com/blog_files/regex.tool.for.emacs.p... and it's good, but not as well integrated. I wish I knew a better way to get a survey of what's available in the Emacs world - something that would have clued me in to this command, for example... any tips? Edit: I want to comment on this as well: learning a lexer/parser is the next step Recently I've had occasion to write a few parsers. In the past, I'd always used parser generators (yacc and antlr) on the assumption that they made things easier. For this project that wasn't an option, so I bit the bullet and did the recursive descent thing. To my surprise, it turned out to be way, way easier than I expected. Moreover, in at least one case (a complex one), the hand-written parser code turned out shorter than the (cl-) yacc version, as well as handling more cases and reporting errors better. The morals of the story are: (1) Don't assume something is hard without trying it; (2) It's not hard to write a classical recursive-descent parser and the skill is a lot more useful than you may realize. Wish I'd learned that a long time ago.
- litewulf 18y ago
  That being said, RDPs have quite a bit of boilerplate that may obscure your grammar. (And small bugs in a RDP are sometimes very hard to catch...)
- mark_h 18y ago
  I wish I knew a better way to get a survey of what's available in the Emacs world - something that would have clued me in to this command, for example... any tips? http://planet.emacsen.org/ http://planet.emacsen.org/ often has useful stuff. The emacs wiki (http://emacswiki.org/ http://emacswiki.org/) is often where you end up after googling and has a heap of information (although even it isn't complete!). You could try http://www.emacswiki.org/cgi-bin/wiki/RandomPage http://www.emacswiki.org/cgi-bin/wiki/RandomPage when you're bored :)
- silentbicycle 18y ago
  I second Emacswiki as suggested, but I find quite a bit by searching through the Emacs and Elisp info pages and using M-x apropos / M-x apropos-variable. It doesn't always work though - to find something, you generally need to know what Emacs calls it.
Darmani 18y ago
If I understand what you mean, you actually can balance tags around arbitrarily nested expressions if your regex parser supports recursion. The Ruby Way has an example of using recursive regexes to balance parentheses.
- silentbicycle 18y ago
  Once your regexes are getting that complicated, though, you're probably better off specifying things in terms of a BNF-based parser (or a recursive-descent parser, depending on the grammar).
jimbokun 18y ago
The article seems like a good tutorial for learning regex constructions, with the examples chosen good motivation for learning.

tocomment 18y ago

meh disappointed. Could any of you guys give me one liners for email and URL validation?

briansmith 18y ago
Don't use regular expressions for those things if you can avoid it. See http://www.regular-expressions.info/email.html http://www.regular-expressions.info/email.html. My URL (IRI) validation regex is a very complicated work in progress and depends on being post-processed by a Python function to handle Unicode characters outside the BMP. It has hundreds of lines of unit tests (not shown): unreserved = ur'(?:[A-Za-z0-9\-\._~]|%s)' % ucschar pct_encoded= ur'%[0-9A-Fa-f]{2}' sub_delims = ur"[!$&'()*+,;=]" pchar = r'(?:%s|%s|%s|[:@])' % (unreserved, pct_encoded, sub_delims) dec_octet = ur'(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])' ipv4address = ur'%s\.%s\.%s\.%s' % (dec_octet, dec_octet, dec_octet, dec_octet) h16 = ur'(?:[0-9A-Fa-f]{1,4})' # 16 bits of address represented in hexadecima ls32 = ur'(?:%s:%s|%s)' % (h16, h16, ipv4address) # least-significant 32 bits of address ipv6address = ( u'(?:' + ur'(?:h16:){6}ls32' + u'|' + ur'::(?:h16:){5}ls32' + u'|' + ur'(h16)?::(?:h16:){4}ls32' + u'|' + ur'((?:h16:){0,1}h16)?::(?:h16:){3}ls32' + u'|' + ur'((?:h16:){0,2}h16)?::(?:h16:){2}ls32' + u'|' + ur'((?:h16:){0,3}h16)?::(?:h16:){1}ls32' + u'|' + ur'((?:h16:){0,4}h16)?::ls32' + u'|' + ur'((?:h16:){0,5}h16)?::h16' + u'|' + ur'((?:h16:){0,6}h16)?::' + u')' ).replace(u"h16", h16).replace(u"ls32",ls32) ipvfuture = ur'v[0-9A-Fa-f]+\.(?:%s|%s|:)+' % (unreserved, sub_delims) ip_literal = ur'\[(?:%s|%s)\]' % (ipv6address, ipvfuture) scheme = ur'(?:[A-Za-z][A-Za-z0-9+\-\.]*)' userinfo = ur'(?:%s|%s|%s|:)*' % (unreserved, pct_encoded, sub_delims) reg_name = ur'(?:%s|%s|%s)*' % (unreserved, pct_encoded, sub_delims) host = ur'(?:%s|%s|%s)' % (ip_literal, ipv4address, reg_name) port = ur'[0-9]*' authority = ur'(?:(?:%s@)?%s(?:\:%s)?)' % (userinfo, host, port) segment = ur'%s*' % pchar segment_nz = ur'%s+' % pchar path_abempty = ur'(?:/%s)*' % ( segment) path_absolute = ur'(?:/(?:%s(?:/%s)*)?)' % (segment_nz, segment) path_rootless = ur'(?:%s(?:/%s)*)' % (segment_nz, segment) path_empty = ur'' hier_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_rootless, path_empty) private = ur'[\ue000-\uF8FF]' # see replaceSurrogatePairs for handling of non-BMP chars query = ur'(?:%s|[/?]|%s)*' % (pchar, private) fragment = ur'(?:%s|[/?])*' % (pchar) iri = ur'%s:%s(?:\?%s)?(?:#%s)?' % (scheme, hier_part, query, fragment) segment_nz_nc = ur'(?:%s|%s|%s|@)+' % (unreserved, pct_encoded, sub_delims) path_noscheme = ur'(?:%s(/%s)*)' % (segment_nz_nc, segment) relative_part = ur'(?://%s%s|%s|%s|%s)' % (authority, path_abempty, path_absolute, path_noscheme, path_empty) relative_ref = ur'(?:%s(?:\?%s)?(?:#%s)?)' % (relative_part, query, fragment) iri_reference = ur'(?:%s|%s)' % (iri, relative_ref)

BrandonM 18y ago

The first tip is okay, but the latter 4 are just horrible. Any decent language will have libraries for parsing HTML or e-mails addresses. A regex is sure to come up short and be very fragile. I once had to maintain some screen-scraping code that was written in Python using regular expressions. By the time I inherited it, half of the functionality no longer worked. It would have been much better off using a library like BeautifulSoup, both in terms of development time and maintainability. BeautifulSoup alone takes care of REs 2 and 3, and there are standard libraries in Python that take care of 4 and 5. Why reinvent (less robustly, I might add) the wheel when a simple API already exists in many languages?

Erwin 18y ago

I wouldn't use a single regexp for complete username validation -- if it fails, all you can display is a generic "username is not valid, it must obey rules X, Y & Z" message. I'd check min and max length separately and display an appropriate error message for that. Also ignore leading/trailing spaces; or you'll end up puzzled why you have two "bob@example.org" users in your database even with appropriate database constraint, and bob mails you saying he can't login on the account he just paid for.

antirez 18y ago

5 Regular Expressions Every Web Programmer should now how to write... if he claims I knows regexps. The problem is that in my experience a minimal part of web developers really know regexps.