14 ms·
Regex for Noobs – An Illustrated Guide
- chupa-chups 7y agoA small collection of gotchas to regexes: https://www.rexegg.com/regex-gotchas.html https://www.rexegg.com/regex-gotchas.html
- manjana 7y agoRegular expressions may appear intimidating to the complete newbie, but once you grasp a few simple ideas I'd argue it is one of the easier topics in CS.
- Biganon 7y agoAnd incredibly useful. I don't know if that's just me and the things I do with a computer, but I feel like it's a knowledge I've used a million times in my life, both for my pet projects and my job as a law school TA. What baffles me is that it's a niche knowledge. Excel is insanely useful, but it's considered a basic knowledge, or at least everyone has heard of it. Not regex.
- gmac 7y agoAgreed. I teach a session on it every year to Economics PhDs and faculty — if I can only get them to come. It's something they just don't know they don't know. http://mackerron.com/text/text-slides.pdf#page19 http://mackerron.com/text/text-slides.pdf#page19
- jacobevelyn 7y agoWow, I just wanted to say thanks for sharing this. These slides are great. Would you mind if I shared this link with some coworkers?
- gmac 7y agoShare away!
- nodesocket 7y agoVery nice write up, and easy to read. The author does a great job explaining the concepts. I do wish he would have dove a bit deeper though: > [0-9][0-9]* (what this pattern matches is left as an exercise to the reader) Some more real world examples to bring all the concepts home would have been great.
- rmason 7y agoI'd also like to see a part 2 and perhaps a part 3. This is the clearest explanation of regex that I've seen - and I've seen quite a few!
- noxer 7y agoThe pattern is kinda nonsense. The first part [0-9] matches one digit in the given range 0-9. The second part obviously the same but the * quantifier means the set can occur 0 or many times. The whole pattern therefore means one digit 0-9 followed by zero or many digits in the range 0-9. so "3" would match "21" would match "123456" would match "" would not match The obviously "correct" pattern for this behavior would be [0-9]+ The + means 1 or more and thus makes repeating the set obsolete.
- Rainymood 7y agoAuthor here. I actually did not know this. I used this very old book (Unix Power Tools) and after finishing the chapter on regex I didn't recall seeing a `+` operator (maybe I forgot to add it or skipped over it). But indeed the `+` operator seems to make significantly more sense here. Thanks!
- SanchoPanda 7y agoWonderful intro, among the best if not the single best basic introductions I've seen.
- Rainymood 7y agoAuthor here, thanks a lot. Your comment means a lot to me!
- sandgraham 7y agohttps://regexone.com/ https://regexone.com/ is a great resource for learning regex. Also try https://regex101.com/ https://regex101.com/ as a sandbox for experimenting and expanding your regex skills.
- Klathmon 7y agoRegex101 is basically my "regex editor", to the point that I have a keybinding in my normal code editor that opens it in the browser for me. It's such a good resource for understanding and writing both simple and complicated regex.
- mettamage 7y agoWhy not use sublime as your regex editor? What makes regex101 special?
- m4tthumphrey 7y agoCan't speak for Sublime but 101 has inline highlighting and explanations per regex feature/flag/pattern. It really is awesome.
- nerdponx 7y agoIt's like the difference between writing C++ in Nano and writing C++ in Visual Studio.
- sverhagen 7y agoI want to call out https://www.regular-expressions.info/ https://www.regular-expressions.info/ since it was, for a long time, one of the better/best resources on regexes that I was able to find. I learned a lot from this ... guy, essentially.
- nerdponx 7y agoThis website is great and well worth a donation if you've benefited from it in the past. As a "next level" site, also give https://rexegg.com https://rexegg.com a look.
- gilboise 7y agoI love the style of the pencil highlighted handwriting!
- Rainymood 7y agoAuthor here. Thanks a lot for the kind words, they really do mean a lot to me :)
- obtino 7y agoThe trouble with regular expressions is not their use, but the different "standards" (or lack thereof) and figuring out what the interpreter supports.
- Biganon 7y agoYes, two issues are a) knowing what to escape and b) what is supported or not. I was weirdly surprised to discover that there are some useful functionalities that are basically supported nowhere... except in vim. I think it was variable-length lookbehinds but don't quote me on this.
- ygra 7y agoLookbehind is restricted in many regex engines that support it (many don't at all). Some require the lookbehind to be constant-length (so if you have alternatives in there, they all have to have the same length, and you can't use quantifiers, basically). Some require it to be finite-but-known-ahead-of-time-length, so something like (?<=a{3,6}) is okay, but (?<=a{3,) is not. Also (?<=a|bb) would be okay. .NET's System.Text.RegularExpressions.Regex is one implementation that has no restrictions on lookbehind. Having used PowerShell for so long it now happens sometimes that I forget when writing regexes for other implementations, as it's really convenient at times.
- asicsp 7y agoyeah, typically you would get error if you use variable length lookbehinds a few do support it (for ex: Python's 3rd party regex module) and sometimes you could workaround with \K (similar to \zs in vim) [1] And there are other frustrating differences between implementations, for example \g definition is very different between Ruby and Python, character set operations are not found everywhere, etc. Plus, BRE/ERE versions found in command line tools do not even support features like non-greedy and lookaround [1] https://stackoverflow.com/questions/11640447/variable-length-lookbehind-assertion-alternatives-for-regular-expressions https://stackoverflow.com/questions/11640447/variable-length...
- chupa-chups 7y agoSo true: https://stackoverflow.com/questions/4644847/list-of-all-regex-implementations https://stackoverflow.com/questions/4644847/list-of-all-rege...
- hartator 7y agoI love https://rubular.com https://rubular.com
- danielparks 7y agoHas anybody seen an introduction to regexes that starts with finite automata? I was introduced to them that way and it seemed very intuitive to me. I’m not sure if I just have a head for regular expressions, or if the way they were presented was especially good.
- tsumnia 7y agoNC State's second semester CS course uses a regex model as an example to Finite State Machines; however, I think it's only an example, not a deep dive into it
- MrFlynn 7y agoMy university (UC Riverside) does this. I haven't taken that course specifically as it's not required for my major, so I can't speak to any specifics.
- blondin 7y agoi remember seeing some japanese anime-style programming guides but never thought anything of them back then. i really miss these short illustrated guides but i don't recall what company was publishing them.
- bane 7y agoI've taught a few "intro to regex" classes and have settled on a very simple explanation that seems to work even with fairly non-technical audiences. In short: 1) Regular expressions and "regex" are different things. Regex is a superset of regular expressions. But if you can understand regular expressions you can understand a huge number of regexes and have the tools to understand the rest of the syntax that the system uses. 2) There are many many many regular expression and regex dialects and engines. It's useful to learn the dialect you are trying to use. 3) Originally, regular expressions were used to describe a generator of strings. It can be helpful to approach writing regexes from this standpoint. It's not about what it matches, but what strings the regex can produce. I've had many students go from bewildered incomprehension to immediate eureka when I describe this. 4) There are only three rules that the beginning regex user has to know: i) Concatenation - you can make a bigger generator by combining one generator after the other. For example: the regex 'a' and the regex 'b' can be concatenated into 'ab' and produces only the string 'ab'. ii) Alternation - you can specify one character or another using the pipe character in most dialects '|'. For example the regex 'a|b' produces either the string 'a' or 'b'. iii) Repetition - the character ' * ' is a postfix operator that means that the character that preceeds it can be repeated zero to an infinite number of times. For example: 'a* ' produces the empty string, 'a', 'aa', 'aaa' and so on. 5) By combining these (i.e. using concatenation) and the grouping operators '(' ')' you can write a regex for almost anything. 6) Nearly all of the other character you see in regexes that don't make sense are simply syntactic sugar that make it easier and shorter to write common expressions in a more compact form. Examples: Writing lots of alternations of single characters (called character classes) - 'a|b|c|d|e|f' can be written as [a-f] - an expression that is the alternation of every character except the newline character can be written as '.' There's a whole host of these kinds of things. Writing lots of different kinds of repetitions introduces lots of new operators - 'aa* ' can be written as 'a+' where '+' means 'repeat the previous character 1 or more times - 'a?' means 'repeat the previous character 0 or one times' - 'a{1,5}' means 'repeat the previous character 1 to 5 times' There's also an entire table of these things. After that it's mostly practice and referring to the description of the operators for the dialect you are using. I usually cover capture groupings (e.g. \1, \2, \3 or $1, $2, $3) later on if necessary as it requires people to have a good handle on the particulars of their languages way of handling those things. At this time I'll also over the '(?:.)' non-capture group operator if their dialect allows. About as frequently I'll have to introduce '^' and '$' and begin and end string operators and also deconflict it from '[^.]'. All the crazy-ass back/forward reference stuff I usually leave out as people almost never need or come across those except in pretty rare instances. In those cases they usually have enough regex under their belt that it's just another topic to pick up. This doesn't get anybody to pro-wizard level and there's always edge cases and weird Perl-golf stuff that's out there, but it'll get most people to generally functional inside of a day or two and I'll just make myself available to spot answer questions or remind them where they forgot something.
- Ocerge 7y agoRegex is one of those topics I just can't seem to commit to memory. The concept is easy enough, and every time I need one I can usually work out what to do after a short amount of research, but I feel like I forget immediately afterwards. It may have to do with how regexes of any complexity look like machine code to me.
- hairui 7y agoThis is a new and excellent way to actually remember how to write regex https://www.executeprogram.com/ https://www.executeprogram.com/ Spaced repetition works!
- ericmcer 7y agoThat was how I was as well until I got tasked with making feature rich text inputs (decorate hashtags, user tags, replace names with emails, vice versa). At a certain point it gets too difficult to respond to user input as it is just too variable (user highlights 40 chars and presses ‘A’? user pastes 100 chars?). A bulletproof method is running find and replace when needed and the ideal way to do that is regex. Forcing myself to get tough regex working is what really made me grasp some of the more advanced concepts that unlock regex. It’s mainly just that we never need regex, and if you use it for a simple task you’re a jerk for introducing unnecessary complexity. Why would people learn it?
- hackbinary 7y agoPractice, my friend. You might have developed a systematic methodology for quickly learning and internalising (many) other things. (Kudos.) But sometimes it just comes down to wrote practice and repetition; time and effort might be key here for your mastery of this subject.
- KronosNull 7y agoThe problem I have with learning regex is similar to the above. It's easy enough to (re)learn when you need it and it doesn't come up often enough for me to consider actually learning it as being useful.
- 7y ago
- qwerty456127 7y ago"sufficiently advanced enough"? Did Arthur C. Clarke really say it this way?
- j88439h84 7y agoThe handwriting is not easy to read.
- Rainymood 7y agoAuthor here, thanks for the feedback. I'll make sure the handwriting is clearer next time. Thank you!
- ktpsns 7y agoI learnt regex when I started with PHP in the 2000s and subsequently when I learned Perl. When I started studying copmuter science at the university, I was supper baffled that there is a full theory of regular expressions: The one of a regular (formal) language (https://en.wikipedia.org/wiki/Regular_language https://en.wikipedia.org/wiki/Regular_language). I recommend everybody who's interested in "looking behind the curtains" of regexp, their powers and limitations, to dive in a crash course of formal languages (Chomsky hierarchy). It can be useful to gain a feeling what a regexp can do and what it cannot.
- mlevental 7y agothe Chomsky hierarchy isn't the actual relevant hierarchy for formal languages - it's the polynomial complexity hierarchy https://en.wikipedia.org/wiki/Polynomial_hierarchy https://en.wikipedia.org/wiki/Polynomial_hierarchy
- Waterluvian 7y agoI've never memorized any regex. It's so syntax dense without many intuitive characters. Not sure this is a problem. They're basically write only. Please always comment beside regexes what they intend on doing.
- spraak 7y agoAlso very helpful to have at least a few good unit tests to show what they should be doing
- aniham 7y agoThrow in + and I think this covers close to 100% of things I've ever had to grep. Very nice little tutorial/refresher.
- thrwway3473 7y agoNice resource. You do start off making it a bit daunting: >For most people without a formal CS education, regular expressions (regex) can come off as something that only the most hardcore unix programmers would dare to touch. my experience wasn't as daunting as this at all. this is my story: basically every regex I ever wrote always worked the first time and I found it super easy. my intro was the Perl 5 "camel" book, i.e. Programming Perl. never heard of regexes before that. If you find this current tutorial we're discussing tricky or daunting, maybe give the resource I just mentioned ("Programming Perl" for Perl 5) a go because for whatever reason the explanation was super simple and writing a regex was one of the easiest things I ever did no matter what I wanted to do. I didn't know about the concept itself before I read that book. I've used it to parse loads and loads of things, use it in my editors, etc. It's always so easy. I don't know if you can find the "Programming Perl" book, I tried to look and found something like that here: https://www.cs.ait.ac.th/~on/O/oreilly/perl/prog/ch01_07.htm https://www.cs.ait.ac.th/~on/O/oreilly/perl/prog/ch01_07.htm you can maybe ignore the stuff about Perl and still sort of follow the tutorial. just an alternative in case someone had their curiosity piqued by this illustrated guide, but it doesn't quite "click" for them,, and wanted to see the same thing written out in another teaching way. And a note that for me it was always not daunting at all!
- Ciberth 7y agoBy any chance studied in Belgium, Ghent? Just curious for educational purpose :)
- thrwway3473 7y ago>By any chance studied in Belgium, Ghent? No, I didn't. (at the time, bought the Perl book in the U.S., where I read it.)
- theclaw 7y agoI also learned regex from that book and had the same experience as you, so although I can't remember how that book worked it's magic (it's been nearly 20 years) I can corroborate your recommendation. I also found the various perldoc pages on regex to be very helpful resource that I referred to frequently back when I was writing perl. Notable pages are perlretut [0], which serves as an intro to regex, perlre [1] which goes into considerable depth, and perlreref [2] which is a handy quick reference for day-to-day work. [0] https://perldoc.perl.org/5.30.0/perlretut.html https://perldoc.perl.org/5.30.0/perlretut.html [1] https://perldoc.perl.org/5.30.0/perlre.html https://perldoc.perl.org/5.30.0/perlre.html [2] https://perldoc.perl.org/5.30.0/perlreref.html https://perldoc.perl.org/5.30.0/perlreref.html
- senthil_rajasek 7y agoWord of caution: Regex is hard to maintain. They are harder to (re)read than C-Style languages.
- ridiculous_fish 7y agoRegex can be as easy to maintain as code, if you treat them like code: 1. Don't put it all on one line 2. Indent nested constructs 3. Write comments See for example Python's verbose regex syntax: https://docs.python.org/3/library/re.html#re.VERBOSE https://docs.python.org/3/library/re.html#re.VERBOSE
- senthil_rajasek 7y agoThe verbose mode helps a bit. I did a quick check and C# seems to have tricks to accomplish "verbose" RE. But still, if you have to comment RE to improve readability is that really an improvement or one more thing to maintain? My preference is to avoid RE in larger code bases.
- radium3d 7y agoI can't memorize regex completely but my favorite cheat sheet for regular expressions is this simple one: https://github.com/ashchan/cheatsheets/blob/master/misc/regular-expressions-cheat-sheet-v2.pdf https://github.com/ashchan/cheatsheets/blob/master/misc/regu... For more advanced expressions I just do a quick google search for existing solutions before attempting to recreate the wheel.
- daolf 7y agoNice write-up, would love something like this for more advanced concepts such as lookahead and condition.
- cusack 7y agoBest regex intro I’ve read
- carstenhag 7y agoIf someone wants to practice doing regexes a bit, you can try out regexcrossword.com or my app (based on the site's idea, but I'm not affiliated to them) https://play.google.com/store/apps/details?id=de.chagemann.regexcrossword https://play.google.com/store/apps/details?id=de.chagemann.r... Note: this is a new account with my real name, I used to have an account here already.
- mrcactu5 7y agohttps://regexper.com/ https://regexper.com/
- anon1253 7y agoRegex named capture groups are somewhat of an underused feature I find, especially with things like simple parsers they help to keep stuff readable https://regular-expressions.mobi/named.html https://regular-expressions.mobi/named.html
- pvorb 7y agoThe same is true for free-spacing mode and comments, which can improve the readability of regular expressions a lot: https://www.regular-expressions.info/freespacing.html https://www.regular-expressions.info/freespacing.html
- theclaw 7y agoI consider myself to be fairly well versed in regex having used it for years, but a recent task to match only stored procedure definitions including the optional block comments immediately preceding them (if any) in large SQL scripts proved completely exasperating. I was deep in the weeds trying to get it to do optional positive look-behind assertions before I just gave up and wrote some code to do it. The gory details of how to just match block comments are best outlined by this blog post detailing one kindred spirit's journey through the wilderness: https://blog.ostermiller.org/find-comment https://blog.ostermiller.org/find-comment, although this wasn't particularly helpful to my situation, it was reassuring to know that I was not alone and might prove interesting to the crowd here.
- nineteen999 7y agoOne thing I still like about Perl is how well the regex syntax is integrated into the language, combined with being installed in just about every Linux modern Linux system, makes it really accessible from your fingertips. If you're in a hurry and just need to do some one-off slicing and dicing, there's not much more powerful than being able to say, eg: $data = "foobarfoo"; $data =~ s/bar/baz/g; print "$data\n"; versus python's more dramatic and formal: import re data = "foobarfoo" data = re.sub('bar', 'baz', data) print data For some reason, the Perl =~ notation sticks in my head much more easily than the Python method. Of course you can string a bunch of sed commands together in a shell script as well, but I find that becomes unwieldy pretty quickly in a lot of situations.
- megaframe 7y agoAgreed. In other languages a part of me always feels like if I'm using regex I'm doing something wrong, because I used it so liberally in Perl. It feels like a Perl-ism so I try to make a conscious effort not to carry those over as they tend to be the source of inefficiencies.
- masklinn 7y agoTBF for this specific case it's very much unnecessary to break out regexes: most languages do have some sort of literal replacement string API: str.replace = replace(...) S.replace(old, new[, count]) -> str Return a copy of S with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced. (though the documentation turns out more misleading than re.sub's: this states it replaces all occurrences but really replaces all non-overlapping occurrences, which re.sub actually spells out properly).
- megaframe 7y agoOddly enough I was asked once to determine if there were any differences between Perl Regex and Python's implementation in an effort to convert a tool I wrote into Python (so other developers could help) I found only one very weird edge case worth noting, in Perl the regex "s/ a* / x /g" (spaces added to prevent formatting) will turn the string "bac" into "xbxxcx" because the "a" technically means zero or more so it matches the spaces between the characters, not so in python it creates the string "xbxcx" because it matched the a in between one time and didn't count the empty spaces. Slightly less accurate results since * does mean zero or more so the space between counts as zero characters.
- Endy 7y agoWell, this is a very good way of teaching people how to double their problems effectively.
- asicsp 7y agoContext: Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. — Jamie Zawinski See also: https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/ https://blog.codinghorror.com/regular-expressions-now-you-ha...
- asicsp 7y agoShameless plug: I've written books [1] which focuses entirely on the specific flavor supported by the tool or the programming language. I use plenty of examples to present the features one by one. These can be used for learning purposes as well as handy reference guide. I think of regular expressions as a mini-programming language, and use loose fitting analogies Anchors --> adding if conditions Alternation --> Conditional OR a(b|c)d = abd|acd --> a(b+c)d = abd+acd Quantifiers --> string repetition and range operators Dot metacharacter + quantifiers + alternation --> Conditional AND Capture group and backreference --> variable Subexpression call --> function Character class --> sets Lookarounds --> custom conditionals Flags --> like command line options So far, I've written books for Ruby and Python regular expressions, and BRE/ERE/PCRE/Rust for "GNU grep and ripgrep". Currently writing a book for GNU sed. [1] https://learnbyexample.github.io/books/ https://learnbyexample.github.io/books/
- slimypi 7y agoNice guide. I would have liked a little more detail. But this is great as it is! Thanks
- stOneskull 7y agoIf there's a 'suggestion box', I suggest to make part 2 about greediness