9 ms·

Learn Regex the Easy Way

323 points by shubhamjain 9y ago

modalduality 9y ago
Good article (didn't realize there were other kinds of lookaround), but maybe the bottom should link to well-tested standards-based regexes instead. URL: ^(((http|https|ftp):\/\/)?([[a-zA-Z0-9]\-\.])+(\.)([[a-zA-Z0-9]]){2,4}([[a-zA-Z0-9]\/+=%&_\.~?\-]*))*$ I recently encountered a case where a URL had an underscore at the end of a subdomain name. It seems underscores are okay anywhere else, but while my friend on Windows was able to load the website, I wasn't (on Linux) using Firefox, curl, remote screenshot service which presumably ran Linux etc. According to various RFCs, they should be okay anywhere within the subdomain name. Has anyone encountered this behavior? Couldn't find anything on the internet; maybe it's just my computer?
- thinkMOAR 9y ago
  hmm interesting, do you still have the domain/url? You could search for it in your history using regex :)
  - junke 9y ago
    https://xkcd.com/1313/ https://xkcd.com/1313/
- sambe 9y ago
  Yes, I feel that the "Bonus" section (with no explanation even) is rather encouraging beginners to mis-use regular expressions in general, and - more specifically - contains errors.
- keeperofdakeys 9y ago
  It seems to mostly come down to differences in how things are defined. DNS itself can handle almost arbitrary data https://tools.ietf.org/html/rfc2181#section-11 https://tools.ietf.org/html/rfc2181#section-11, while an Internet Hostname was defined to be more strict https://tools.ietf.org/html/rfc1123#section-2 https://tools.ietf.org/html/rfc1123#section-2. The same issue also exists with dashes at the end of domain components. I'm not enough of a history boffin to know how Microsoft came to support it differently (perhaps something from the Netbios and NT era). At this point in time though, I don't see either party changing their default validations to agree on a single definition. Edit: If you're curious, this is the first commit that appears to be the first glibc commit limiting dashes at the end of URLS https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fa0bc87c32d02cd81ec4d0ae00e0d943c683e6e1#patch31 https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fa0bc.... I don't know about BSD libc, or windows however.
  - modalduality 9y ago
    Wow, how did you find that commit?
    - keeperofdakeys 9y ago
      Both git and this git web view allow you to view all the commits that have modified just that file. Eg. https://sourceware.org/git/?p=glibc.git;a=history;f=resolv/res_comp.c;h=1b47a5dfbe6bdbc24904010970d2ecfe914a7812;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=history;f=resolv/r.... So it's a simple matter of looking at the diffs between commits. Of course that's assuming you know the right file, which is often the harder problem.
  - tzs 9y ago
    Wait a second...does this imply that if I put downloads that should only be of interest to our Windows customers on a server named something like downloads_.ourdomain.com, it might keep out all those annoying bots that ignore robots.txt and make a lot of noise in my logs? I'm guessing that most of the bots are not running on Windows.
    - keeperofdakeys 9y ago
      That's a pretty bad idea, you shouldn't rely on this kind of stuff. If there are people running OSX or Linux that want Windows downloads, or someone is behind a captive portal or proxy (like squid), they probably won't be able to reach it anymore. If you have a real problem with bots, I'd look at what IPs they are coming from, and how often they try to connect. Something like IP blacklisting, or fail2ban might work for your use case.
- Sir_Cmpwn 9y ago
  I personally avoid regexes where possible, including in this situation. IMO the right way to validate a URL is to feed it to a URL parser and see if it errors out. I can see errors in this regex right away - and in many other regexes you find from Googling. People just drop them into their codebase and their eyes glaze over when you ask them whether or not it's actually correct. How many websites fail on user+whatever@gmail.com because they copied a bad regex?
VeejayRampay 9y ago
Very good post. It has all the important information, provides clear examples, doesn't try to get too fancy or showboat. Well done.
jules 9y ago
I didn't truly understand regular expressions until I saw how they are executed. There are simple algorithms for executing them, so that might not be such a bad way to teach.
- hackermailman 9y ago
  I learned from the book The Unix Programming Environment but I also didn't truly understand regexp until I read how they were executed, in Programming in Standard ML the first chapter shows how to implement a complete package/parser for regexp http://www.cs.cmu.edu/~rwh/isml/book.pdf http://www.cs.cmu.edu/~rwh/isml/book.pdf
maddyboo 9y ago
My biggest issue with regular expressions is remembering the exact syntax of each of the different common regex engines. Javascript, Perl, `grep`, `grep -E`, vim, `awk`, `sed`, etc... Each one seems to have slightly different syntax, require different characters to be escaped, has different defaults (is global search enabled by default? Multiline? What about case sensitivity?), some don't support certain lookarounds, how does grouping work, and so on.
- deleted 9y ago
  [deleted]
- masklinn 9y ago
  Also replacement (sed, emacs, idea, …), is it $1, is it \1, is it something else, does it support named groups?
- davidreiss 9y ago
  That's one of my pet peeves too. Another pet peeve is that I only touch regex once every few months. When you have a regex set to parse a domain of data you move on to other things. After a while you forget the regex syntax and when you have to parse another set of text or have to debug the older regex, I end up having to relearn a bit of it. But I guess it comes with the territory.
  - ridiculous_fish 9y ago
    Python has a nice trick for commenting regexs with re.VERBOSE. I use it for all non-trivial regexs. https://docs.python.org/2/library/re.html#re.VERBOSE https://docs.python.org/2/library/re.html#re.VERBOSE
- sglane 9y ago
  There are three types of regex as far as I know: basic (aka GNU), extended and PERL. Grep uses GNU as the name implies. egrep or grep -E uses extended. PERL is used elsewhere like JavaScript. Typically you'll see pcre which is the library for Perl Compatible Regular Expressions.
j05huaNathaniel 9y ago
Could use some work explaining capture groups
- com2kid 9y ago
  Capture groups are where I get 99% of value from regexs. Being able to quickly transform data is where I find regexs perform best. Just matching is not something I have to do all that often.
chenster 9y ago
For email, regular expression, there's http://emailregex.com http://emailregex.com
chenster 9y ago
A picture is worth a thousand words. How about a visual regex tester - http://emailregex.com/regex-visual-tester/#a%5Cbc%5Cd* http://emailregex.com/regex-visual-tester/#a%5Cbc%5Cd*
gmac 9y ago
For an even-more-beginner's guide, see the slides to a session I teach to Econ postgrads once a year[1]. These introduces the metacharacters gradually, using a task-based approach. We start by finding street addresses, per https://xkcd.com/208/ https://xkcd.com/208/. [1] http://mackerron.com/text/text-slides.pdf http://mackerron.com/text/text-slides.pdf (page 19 onwards) with supporting resources at http://mackerron.com/text/ http://mackerron.com/text/
clement75009 9y ago
For me, the best Regex ressource is still http://regexr.com http://regexr.com It explains what each character does just by hovering over a regex. Best tool to learn or to fine tune your regular expression (with testing included).
- warcode 9y ago
  I'll add https://regex101.com https://regex101.com as an alternative.
- chrisan 9y ago
  This is a paid app, but the debugger on this has been my favorite https://www.regexbuddy.com/ https://www.regexbuddy.com/ You can even step through the matching process and see how your matches are made
  - squeaky-clean 9y ago
    Love RegexBuddy. That feature along with the "use" tab which generates code for whatever language you select. I don't have to remember the specifics in all the languages I work in. I just select from the dropdowns, for example, "JavaScript (Chrome)" then "Use regex object to get the part of a string matched by a numbered group". Replace the placeholder variable names, and you're good to go! It will also do things like warn you if you use named groups if your selected language doesn't support them, and the "Use" dropdown won't provide that option. I really wish it wasn't Windows only.
    - chrisan 9y ago
      Hah wow... I've had this app for over a decade and never noticed that feature... right next to the Debug button which I have used numerous times on really gnarly regexes
- hacker_9 9y ago
  I'll add http://www.ultrapico.com/expresso.htm http://www.ultrapico.com/expresso.htm as another alternative! Writes out the regex steps in English too.
reificator 9y ago
I spent some time reading some resource or another on how regexes work, but the vast majority of my learning has been trying things in https://regex101.com/ https://regex101.com/ and seeing if they do what I want. The breakdown on the side of the page is especially helpful.
retox 9y ago
Waiting for the additional "Read someone else's Regex the easy way". I'm not holding my breath :) Agree with others in that RegexBuddy is indispensable for a windows dev learning this magic stuff. Some useful and interesting regex developments coming in the next version of JavaScript. Support for international text and (bleh) emoji incoming.
reuven 9y ago
I've been teaching regular expressions for years, and offer a free online e-mail course on the subject (http://RegexpCrashCourse.com/ http://RegexpCrashCourse.com/). This site is a very nice summary of regexp syntax and is written well -- but it's missing two crucial pieces that help people learn: Examples and exercises. Without practice, there's no way that people can remember the syntax.
- darkstar999 9y ago
  It isn't missing those. Every section links to regex101.com prefilled, ready for experimentation.
  - reuven 9y ago
    I'll revise my criticism, then: There are some examples, but not enough. And the links to an interactive regexp system is very smart and nice.
frou_dh 9y ago
I learned regex the "Ambient" way. i.e. Encountering them here, there and everywhere. Then one day realising you have a good knowledge of the subject without ever having set out to learn it.
crncosta 9y ago
I really enjoy this type of tutorial format, concise and easy to follow. Thanks for take time to produce it.
Willamin 9y ago
The lack of readability of regex makes me wonder if there isn't a better way. I've seen Elm's parser which introduces a few neat concepts like parser pipelines. https://github.com/elm-tools/parser https://github.com/elm-tools/parser
- mclehman 9y ago
  Have you looked at Perl 6 at all? Whitespace in regexen is insignificant if not quoted, so not only can you add a little space between sections, you can split a regex over several lines and add comments throughout. It also has first-class grammars, so you're less tempted to reach for regex when something more powerful would help.
gregmac 9y ago
For some people Regex Golf [1] might be an interesting way to learn. You are actually building increasingly complex regex as you go, and can just look up bits of syntax you don't know as needed. [1] https://alf.nu/RegexGolf https://alf.nu/RegexGolf
jwilk 9y ago
Previously: https://news.ycombinator.com/item?id=14846506 https://news.ycombinator.com/item?id=14846506 I'm afraid not much has been improved since then. This is not a good learning source.
dayvid 9y ago
https://regexone.com/ https://regexone.com/ helped me finally learn Regex in 2-3 hours. It's a step-by-step interactive site. One of the best educational programming sites I've been to. Testing Regex is a lot easier when you have the fundamentals down and there's a million resources to test Regex.
Groxx 9y ago
And when you think you've learned regex, learn that you haven't: http://fent.github.io/randexp.js http://fent.github.io/randexp.js (a regex "reverser" of sorts) [1] Seriously. Test every non-trivial regex with something like this, you'll probably be surprised at how permissive most regexes are. Regexes are great. They're super-concise and perform amazingly well. But they're one of the biggest footguns I know of. Treat them as such and you'll probably do fine. --- [1] for instance, the URL regex they use is incorrect, and it's super obvious when you plug it into that site: ^(((http|https|ftp):\/\/)?([[a-zA-Z0-9]\-\.])+(\.)([[a-zA-Z0-9]]){2,4}([[a-zA-Z0-9]\/+=%&_\.~?\-]*))*$ `[[a-zA-Z0-9]\-\.]` you can't nest character sets like that. So this matches the letters "[]-." as well as all a-z,A-Z,0-9 ranges.

323 points by shubhamjain 9y ago

Good article (didn't realize there were other kinds of lookaround), but maybe the bottom should link to well-tested standards-based regexes instead. URL: ^(((http|https|ftp):\/\/)?([[a-zA-Z0-9]\-\.])+(\.)([[a-zA-Z0-9]]){2,4}([[a-zA-Z0-9]\/+=%&_\.~?\-]*))*$ I recently encountered a case where a URL had an underscore at the end of a subdomain name. It seems underscores are okay anywhere else, but while my friend on Windows was able to load the website, I wasn't (on Linux) using Firefox, curl, remote screenshot service which presumably ran Linux etc. According to various RFCs, they should be okay anywhere within the subdomain name. Has anyone encountered this behavior? Couldn't find anything on the internet; maybe it's just my computer?

thinkMOAR 9y ago
hmm interesting, do you still have the domain/url? You could search for it in your history using regex :)
- junke 9y ago
  https://xkcd.com/1313/ https://xkcd.com/1313/
sambe 9y ago
Yes, I feel that the "Bonus" section (with no explanation even) is rather encouraging beginners to mis-use regular expressions in general, and - more specifically - contains errors.
keeperofdakeys 9y ago
It seems to mostly come down to differences in how things are defined. DNS itself can handle almost arbitrary data https://tools.ietf.org/html/rfc2181#section-11 https://tools.ietf.org/html/rfc2181#section-11, while an Internet Hostname was defined to be more strict https://tools.ietf.org/html/rfc1123#section-2 https://tools.ietf.org/html/rfc1123#section-2. The same issue also exists with dashes at the end of domain components. I'm not enough of a history boffin to know how Microsoft came to support it differently (perhaps something from the Netbios and NT era). At this point in time though, I don't see either party changing their default validations to agree on a single definition. Edit: If you're curious, this is the first commit that appears to be the first glibc commit limiting dashes at the end of URLS https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fa0bc87c32d02cd81ec4d0ae00e0d943c683e6e1#patch31 https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=fa0bc.... I don't know about BSD libc, or windows however.
- modalduality 9y ago
  Wow, how did you find that commit?
  - keeperofdakeys 9y ago
    Both git and this git web view allow you to view all the commits that have modified just that file. Eg. https://sourceware.org/git/?p=glibc.git;a=history;f=resolv/res_comp.c;h=1b47a5dfbe6bdbc24904010970d2ecfe914a7812;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=history;f=resolv/r.... So it's a simple matter of looking at the diffs between commits. Of course that's assuming you know the right file, which is often the harder problem.
- tzs 9y ago
  Wait a second...does this imply that if I put downloads that should only be of interest to our Windows customers on a server named something like downloads_.ourdomain.com, it might keep out all those annoying bots that ignore robots.txt and make a lot of noise in my logs? I'm guessing that most of the bots are not running on Windows.
  - keeperofdakeys 9y ago
    That's a pretty bad idea, you shouldn't rely on this kind of stuff. If there are people running OSX or Linux that want Windows downloads, or someone is behind a captive portal or proxy (like squid), they probably won't be able to reach it anymore. If you have a real problem with bots, I'd look at what IPs they are coming from, and how often they try to connect. Something like IP blacklisting, or fail2ban might work for your use case.
Sir_Cmpwn 9y ago
I personally avoid regexes where possible, including in this situation. IMO the right way to validate a URL is to feed it to a URL parser and see if it errors out. I can see errors in this regex right away - and in many other regexes you find from Googling. People just drop them into their codebase and their eyes glaze over when you ask them whether or not it's actually correct. How many websites fail on user+whatever@gmail.com because they copied a bad regex?

VeejayRampay 9y ago

Very good post. It has all the important information, provides clear examples, doesn't try to get too fancy or showboat. Well done.

jules 9y ago

I didn't truly understand regular expressions until I saw how they are executed. There are simple algorithms for executing them, so that might not be such a bad way to teach.

hackermailman 9y ago
I learned from the book The Unix Programming Environment but I also didn't truly understand regexp until I read how they were executed, in Programming in Standard ML the first chapter shows how to implement a complete package/parser for regexp http://www.cs.cmu.edu/~rwh/isml/book.pdf http://www.cs.cmu.edu/~rwh/isml/book.pdf

maddyboo 9y ago

My biggest issue with regular expressions is remembering the exact syntax of each of the different common regex engines. Javascript, Perl, `grep`, `grep -E`, vim, `awk`, `sed`, etc... Each one seems to have slightly different syntax, require different characters to be escaped, has different defaults (is global search enabled by default? Multiline? What about case sensitivity?), some don't support certain lookarounds, how does grouping work, and so on.

deleted 9y ago
[deleted]
masklinn 9y ago
Also replacement (sed, emacs, idea, …), is it $1, is it \1, is it something else, does it support named groups?
davidreiss 9y ago
That's one of my pet peeves too. Another pet peeve is that I only touch regex once every few months. When you have a regex set to parse a domain of data you move on to other things. After a while you forget the regex syntax and when you have to parse another set of text or have to debug the older regex, I end up having to relearn a bit of it. But I guess it comes with the territory.
- ridiculous_fish 9y ago
  Python has a nice trick for commenting regexs with re.VERBOSE. I use it for all non-trivial regexs. https://docs.python.org/2/library/re.html#re.VERBOSE https://docs.python.org/2/library/re.html#re.VERBOSE
sglane 9y ago
There are three types of regex as far as I know: basic (aka GNU), extended and PERL. Grep uses GNU as the name implies. egrep or grep -E uses extended. PERL is used elsewhere like JavaScript. Typically you'll see pcre which is the library for Perl Compatible Regular Expressions.

j05huaNathaniel 9y ago

Could use some work explaining capture groups

com2kid 9y ago
Capture groups are where I get 99% of value from regexs. Being able to quickly transform data is where I find regexs perform best. Just matching is not something I have to do all that often.

chenster 9y ago

For email, regular expression, there's http://emailregex.com http://emailregex.com

chenster 9y ago

A picture is worth a thousand words. How about a visual regex tester - http://emailregex.com/regex-visual-tester/#a%5Cbc%5Cd* http://emailregex.com/regex-visual-tester/#a%5Cbc%5Cd*

gmac 9y ago

For an even-more-beginner's guide, see the slides to a session I teach to Econ postgrads once a year[1]. These introduces the metacharacters gradually, using a task-based approach. We start by finding street addresses, per https://xkcd.com/208/ https://xkcd.com/208/. [1] http://mackerron.com/text/text-slides.pdf http://mackerron.com/text/text-slides.pdf (page 19 onwards) with supporting resources at http://mackerron.com/text/ http://mackerron.com/text/

clement75009 9y ago

For me, the best Regex ressource is still http://regexr.com http://regexr.com It explains what each character does just by hovering over a regex. Best tool to learn or to fine tune your regular expression (with testing included).

warcode 9y ago
I'll add https://regex101.com https://regex101.com as an alternative.
chrisan 9y ago
This is a paid app, but the debugger on this has been my favorite https://www.regexbuddy.com/ https://www.regexbuddy.com/ You can even step through the matching process and see how your matches are made
- squeaky-clean 9y ago
  Love RegexBuddy. That feature along with the "use" tab which generates code for whatever language you select. I don't have to remember the specifics in all the languages I work in. I just select from the dropdowns, for example, "JavaScript (Chrome)" then "Use regex object to get the part of a string matched by a numbered group". Replace the placeholder variable names, and you're good to go! It will also do things like warn you if you use named groups if your selected language doesn't support them, and the "Use" dropdown won't provide that option. I really wish it wasn't Windows only.
  - chrisan 9y ago
    Hah wow... I've had this app for over a decade and never noticed that feature... right next to the Debug button which I have used numerous times on really gnarly regexes
hacker_9 9y ago
I'll add http://www.ultrapico.com/expresso.htm http://www.ultrapico.com/expresso.htm as another alternative! Writes out the regex steps in English too.

reificator 9y ago

I spent some time reading some resource or another on how regexes work, but the vast majority of my learning has been trying things in https://regex101.com/ https://regex101.com/ and seeing if they do what I want. The breakdown on the side of the page is especially helpful.

retox 9y ago

Waiting for the additional "Read someone else's Regex the easy way". I'm not holding my breath :) Agree with others in that RegexBuddy is indispensable for a windows dev learning this magic stuff. Some useful and interesting regex developments coming in the next version of JavaScript. Support for international text and (bleh) emoji incoming.

reuven 9y ago

I've been teaching regular expressions for years, and offer a free online e-mail course on the subject (http://RegexpCrashCourse.com/ http://RegexpCrashCourse.com/). This site is a very nice summary of regexp syntax and is written well -- but it's missing two crucial pieces that help people learn: Examples and exercises. Without practice, there's no way that people can remember the syntax.

darkstar999 9y ago
It isn't missing those. Every section links to regex101.com prefilled, ready for experimentation.
- reuven 9y ago
  I'll revise my criticism, then: There are some examples, but not enough. And the links to an interactive regexp system is very smart and nice.

frou_dh 9y ago

I learned regex the "Ambient" way. i.e. Encountering them here, there and everywhere. Then one day realising you have a good knowledge of the subject without ever having set out to learn it.

crncosta 9y ago

I really enjoy this type of tutorial format, concise and easy to follow. Thanks for take time to produce it.

Willamin 9y ago

The lack of readability of regex makes me wonder if there isn't a better way. I've seen Elm's parser which introduces a few neat concepts like parser pipelines. https://github.com/elm-tools/parser https://github.com/elm-tools/parser

mclehman 9y ago
Have you looked at Perl 6 at all? Whitespace in regexen is insignificant if not quoted, so not only can you add a little space between sections, you can split a regex over several lines and add comments throughout. It also has first-class grammars, so you're less tempted to reach for regex when something more powerful would help.

gregmac 9y ago

For some people Regex Golf [1] might be an interesting way to learn. You are actually building increasingly complex regex as you go, and can just look up bits of syntax you don't know as needed. [1] https://alf.nu/RegexGolf https://alf.nu/RegexGolf

jwilk 9y ago

Previously: https://news.ycombinator.com/item?id=14846506 https://news.ycombinator.com/item?id=14846506 I'm afraid not much has been improved since then. This is not a good learning source.

dayvid 9y ago

https://regexone.com/ https://regexone.com/ helped me finally learn Regex in 2-3 hours. It's a step-by-step interactive site. One of the best educational programming sites I've been to. Testing Regex is a lot easier when you have the fundamentals down and there's a million resources to test Regex.

Groxx 9y ago

And when you think you've learned regex, learn that you haven't: http://fent.github.io/randexp.js http://fent.github.io/randexp.js (a regex "reverser" of sorts) [1] Seriously. Test every non-trivial regex with something like this, you'll probably be surprised at how permissive most regexes are. Regexes are great. They're super-concise and perform amazingly well. But they're one of the biggest footguns I know of. Treat them as such and you'll probably do fine. --- [1] for instance, the URL regex they use is incorrect, and it's super obvious when you plug it into that site: ^(((http|https|ftp):\/\/)?([[a-zA-Z0-9]\-\.])+(\.)([[a-zA-Z0-9]]){2,4}([[a-zA-Z0-9]\/+=%&_\.~?\-]*))*$ `[[a-zA-Z0-9]\-\.]` you can't nest character sets like that. So this matches the letters "[]-." as well as all a-z,A-Z,0-9 ranges.