13 ms·
My favorite regex of all time
- lambada 14y agoAre people seriously still deliberately using ASCII-reliant code?
- manojlds 14y agoIt's interesting. Doesn't mean it's worthy of being put in production code.
- easytiger 14y agoit does if the text you are dealing with is specified as ascii only
- yxhuvud 14y agoYes, when dealing with RFC's that do.
- olalonde 14y agoFor file names, URLs, domain names, etc. it's usually the safe thing to do.
- recursive 14y agoNot as often as often as handling all characters.
- UnoriginalGuy 14y agoWho's filenames aren't unicode? Also domains and URLs can be unicode too.
- esrauch 14y agohttp://en.wikipedia.org/wiki/IDN_homograph_attack http://en.wikipedia.org/wiki/IDN_homograph_attack
- olalonde 14y ago> Who's filenames aren't unicode? Many filesystems don't support unicode or support only a subset of it: https://en.wikipedia.org/wiki/Filename#Comparison_of_filename_limitations https://en.wikipedia.org/wiki/Filename#Comparison_of_filenam... > Also domains and URLs can be unicode too. Domains: it depends at which level you are dealing with them. See https://en.wikipedia.org/wiki/Internationalized_domain_name https://en.wikipedia.org/wiki/Internationalized_domain_name Internationalized domain names are stored in the Domain Name System as ASCII strings using Punycode transcription. URLs: Unicode characters are not allowed in URLs. See http://www.faqs.org/rfcs/rfc1738.html http://www.faqs.org/rfcs/rfc1738.html and http://www.blooberry.com/indexdot/html/topics/urlencoding.htm http://www.blooberry.com/indexdot/html/topics/urlencoding.ht... only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.
- IheartApplesDix 14y agoDid ducttape stop being sticky?
- csense 14y agoEvery time I've had to deal with Unicode and internationalization, it's been a problem. For example, a few years ago I grabbed a source tarball from somewhere, I forget what or where. It had the author's name in a comment, which included an O with dots over it. That was the only non-ASCII character in the source code. No matter what I did, both Eclipse and command-line javac refused to compile the source. Finally I wrote a script to delete his name from every source file manually. It compiled flawlessly. Then there's the time I found some text files with two characters of binary junk at the beginning, followed by completely normal text. Again, I forget what I was doing, but some program was refusing to process them correctly. It was something internationalization-related called the BOM. Eventually I ended up writing a script to walk a directory and remove the first two bytes of every file. (This can probably be done with dd and xargs on UNIX, but I was using Windows at the time, which means that something like this will require spending an hour or so in your favorite programming language.) These experiences lead me to believe that, for bootstrapped USA startups at least, you shouldn't worry about a market outside the English-speaking world. If you need to worry about junk like accented characters or moon runes (Chinese/Japanese/Korean characters), it means you're big enough to afford to hire someone specifically to address the problem.
- andrewflnr 14y agoWhat? You suffered from other peoples' bad internationalization, which implies that people shouldn't care about internationalization?
- token_female 14y ago> moon runes (Chinese/Japanese/Korean characters) Really? You can make your economic point without making fun of several of the world's most commonly spoken languages and the people who speak them.
- e12e 14y agoI assume this is a not very subtle troll? Java source is unicode? (The offhand reference to dd and xargs is a bit too much). How do you define "English-speaking world", btw? Those too ignorant to have heard of non-ascii-characters (ie: excluding Canada, as anyone doing business there should at least have heard of French)? Anyway, for anyone actually burnt by something similar on a GNU system try looking up recode(1).
- kamaal 14y agoI think HN is written Arc, which is not very Unicode friendly.
- StavrosK 14y agoΔοκιμή. EDIT: It works fine for comments, at least.
- skrebbel 14y agoIt's clever, but it's also completely unreadable for anyone who didn't read this article. Regexes have serious maintainability issues as it is; let's not make it worse by putting clever tricks in them.
- akdetrick 14y agoEvery regex seems like a clever trick.
- Terretta 14y agoI don't understand why this is "completely unreadable". What else could this have been besides match the character range from space to tilde?
- jlgreco 14y agoPresumably the space, commonly having no meaning in and of itself, could throw you for a moment or two. This isn't a regex `foo_[a-z]`, you have to stop and think about it for a moment. I don't think it is particularly bad though. It's just not the most trivial of regexes.
- mikeash 14y agoMost people would have to check an ASCII table to know what that range is, though.
- slavak 14y agoWhich takes for granted the fact that your input stream is even ASCII to begin with. I'm too lazy to check, but I'm pretty sure this isn't going to catch all printable Unicode characters, for example - and then you're left scratching your head over what the hell the original author was trying to achieve.
- michaelt 14y agoThe main risk in my mind was that it was some sort of control sequence for a feature I hadn't memorized. I know there's some syntax I can use to create a zero width negative look behind recursive greedy named capture group back reference. Perhaps hyphen-tilde triggers something like that.
- jimwise 14y agoThis will not only miss non-ascii printing characters, but it's not even much shorter than typing [[:print:]] to use the explicit character class.
- DrCatbox 14y agoThe [[:print:]] will match any printable characters like åä, while the [ -~] will not. I used this once as another safeguard against pushing binary data into the database. It was a poor system to begin with where you even have that possibility... and it happened at least once before the fix and my safeguard was in place.
- e12e 14y ago"å" is perfectly valid text input in my locale.
- nowarninglabel 14y agoI think that was his point, that he had a good use for :print: over just -~
- EvilTerran 14y agoThere will be situations where you need to check specifically for 7-bit ASCII printable characters only. I've worked with APIs that require everything outside that range to be escaped/encoded into it. Email could be an example, I guess, although I haven't worked with it enough to know whether the whole "7-bits only" thing is still an issue these days.
- IsTom 14y agoThis works for ASCII only, use unicode character classes instead.
- smegel 14y agoWhy?
- csense 14y agoThat only matters if you need to process Unicode. See my comments [1] [2] [3] for why Unicode / internationalization should be avoided. [1] http://news.ycombinator.com/item?id=4369323 http://news.ycombinator.com/item?id=4369323 [2] http://news.ycombinator.com/item?id=4541039 http://news.ycombinator.com/item?id=4541039 [3] http://news.ycombinator.com/item?id=4775440 http://news.ycombinator.com/item?id=4775440
- kolinko 14y agoSo, do you propose that u.s. bootstrapped startups have a disclaimer on the registration page saying: "you cannot put foreign characters anywhere in our system"? Even if you focus on u.s., you will have problems. If you're doing a CRM, even u.s. users will put in foreign names from time to time. If you're building a CMS, users may want to put in a quotation in french, or will simply use copy&paste from Word, which replaces "-" with "—"... I honestly have a hard time finding a u.s. centric startup which could afford to ignore unicode. The support requests, the fires caused by errors, and the disclaimer that you'd have to put on the registration page, would cost much more than simply learning how to code the f'n utf. Building MVP is good practice in Lean. Saying "I'm bootstrapping hence I don't have the time to learn the programming tools" is just ignorance and incompetence. It's not like Unicode gives you extra work, it just requires you to learn a few basic concepts. If you try to build a site which doesn't support Unicode, you'll have to put a lots of safeguards everywhere to cover up for your incompetence.
- noonespecial 14y agoAs someone who makes much of his living rehabilitating old perl scripts, please, if you must use such things, use them like this: [ -~] #match only printable characters It takes 5 seconds longer and with regexes, just knowing what the damn thing is trying to do is half the battle. When you use a regex, use a comment. Its the civil thing to do.
- sbochins 14y agoYea, anytime I use a regex that isn't immediately obvious I put it in a function called get_<something>. Unfortunately people that write overly complicated and error prone regexes usually don't choose to document them.
- laumars 14y agoIf a regex is going to be reusable, then yeah, I'd agree. But dumping single lines of code into their own functions just for readability isn't practical for real time systems. In those cases you really should be using comments as they get stripped out by the compiler.
- entropy_ 14y agoCouldn't those functions just be inlined by the compiler if they're simple regex-wrappers anyway? I do agree that it might be overkill to move regexes to their own functions just for readability's sake but I don't buy the performance argument. Furthermore, regexes are most popular in scripting languages that no sane person would use for real time performance-critical systems anyway.
- laumars 14y ago1. Ahh right. I wasn't aware that happened. 2. Web sites are a classic example of scripting languages being used for real time performance critical systems (though I'm not arguing that all web sites are real time). Sometimes the ability to modify code easily is as important to the choice of languages as the raw execution speed of the compiled binaries.
- ceejayoz 14y agoMy favourite is this one: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html
- jballanc 14y agoAh, this is my favorite also. If seeing this doesn't make you second guess using a RegExp when a parser is more appropriate, well...you might be a Perl programmer?
- l-p 14y agoI think this one is more appropriate: http://nikic.github.com/2012/06/15/The-true-power-of-regular-expressions#matching_contextfree_languages http://nikic.github.com/2012/06/15/The-true-power-of-regular...
- carlio 14y agoThis seems to be a T-shirt advert, why am I reading this on HN?
- jack-r-abbit 14y agoJeepers... cut the guy some slack. He didn't say this is the bullet proof way of doing everything YOU want to do in all situations, every time, forever. He said "I thought I'd share my favorite regex of all time". And then explained what it does. Why does everyone have to poop on his favorite thing?
- boyter 14y agoMy favorite regex is the following, /^1?$|^(11+?)\1+$/ Which finds prime numbers. Although, I can't for the life of me think of a reason for using it. http://stackoverflow.com/questions/3296050/how-does-this-regex-find-primes http://stackoverflow.com/questions/3296050/how-does-this-reg...
- deleted 14y ago[deleted]
- mvzink 14y agoI had to prove that in a formal languages class once and I still have no idea how it works.
- emillon 14y agoFirst, it does not match "prime numbers". It matches composite numbers in unary notation (n is represented by n '1' characters). The first part (^1?$) allows "" and "1" to match (so that 1 is not detected as a prime). The second part matches groups of two or more ones (11+?), repeated twice or more, ie products n*m, n ≥ 2, m ≥ 2. The backreference means that \1 should match the exact same string as the first (11+?). It's different from using (11+?){2,} which would match n_1+n_2+n_3..., n_1 ≥ 2, n_2 ≥ 2, n_3 ≥ 2 (where submatch is independent).
- lubutu 14y agoI do dislike people calling that expression a "regex", because it isn't: regular expressions cannot contain backreferences, and must be computable in linear time, whereas primality tests are polynomial.
- boyter 14y agoWhile I agree I believe this comment by _delirium sums this up rather well, http://news.ycombinator.com/item?id=1486502 http://news.ycombinator.com/item?id=1486502 full comment thread here http://news.ycombinator.com/item?id=1486158 http://news.ycombinator.com/item?id=1486158
- kamakazizuru 14y agowhat a plug - obviously the only real purpose of this was to sell those t-shirts.
- pkrumins 14y agoI'm sorry that was not the purpose. I just wanted to share this regex trick that I had in my mind. I added the shirts only later when I saw that the article is getting very popular. People really seemed to like my previous tees and it's great to make a little extra money and continue doing what I love - coding and writing blog posts.
- bhanks 14y agowell obviously people are interested in it. Can't fault you for trying to make a little scratch
- gpvos 14y agoMy favourite regex is actually: [^ -~] Not to be used in a serious program, but only in an editor (or maybe one-shot data massage perl scripts), to find possible errors or unexpected stuff.
- tathastu 14y agoAlso it's more interesting to put unprintable characters on a t-shirt.
- Mahn 14y ago[\p{L&}] <- unicode version, in case you were wondering.
- e12e 14y agoNo, you don't need '[' and ']' ? Also you don't need the '&' to get the equivalent of the above, but might have to add support for spaces? > \p{L} or \p{Letter}: any kind of letter from any language. vs > \p{L&} or \p{Letter&}: a letter that exists in lowercase and uppercase variants (combination of Ll, Lu and Lt). Along with: > \p{Z} or \p{Separator}: any kind of whitespace or invisible separator. Considering the op matches everything printable, including whitespace (or actually just space, not tab), numbers and punctation, I think the equivalent would be "\X" ? All this based on glancing at: http://www.regular-expressions.info/unicode.html
- sk5t 14y agoI suppose a single regex can be both "favorite" and "worst" at the same time... it's only slightly interesting to know where ~ appears in the ASCII character set, and while someone might recall that space is kinda near the beginning but after the control characters, is it the first helpful printable character? Who knows?
- NegativeK 14y ago> I suppose a single regex can be both "favorite" and "worst" at the same time... We definitely aren't the only ones who appreciate horrible things. INTERCAL comes to mind here.
- unjinxable 14y agoI like to use a variation of this in vim to quickly see if an html doc I'm working on contains weird characters that I might want to replace with &html; entities: /[^ -~]
- jervisfm 14y agoCan anyone explain how this regex [- ~] matches ASCII characters ?
- boyter 14y agoIt's pretty simple. Assuming you know regex... Im going to assume you don't since you are asking. The bracket expression [ ] defines single characters to match, however you can have more then 1 character inside which all will match. [a] matches a [ab] matches either a or b [abc] matches either a or b or c [a-c] matches either a or b or c. The - allows us to define the range. You can just as easily use [abc] but for long sequences such as [a-z] consider it short hand. In this case [ -~] it means every character between <space> and <tilde>, which just happens to be all the ASCII printable characters (see chart in the article). The only bit you need to keep in mind is that <space> is a character as well, and hence you can match on it. You could rewrite the regex like so (note I haven't escaped or anything in this so its probably not valid) [ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~] but that's not quite as clever or neat.
- mutation 14y agoIt doesn't. Space is significant here, and if '-' is a the front of the matching character class it matches literal '-'. Your regex '[- ~]' matches either '-' or ' ' or '~'.
- hsiaobrandon 14y agoI don't want to be "that guy" and it's probably just my own stupidity, but what's so special about this for it to frontpage HN?
- kaokun 14y agoNot all strings are ASCII! :-(
- tripzilch 14y agoWouldn't it be smarter, in that case, match something like [\x20-\x7F] (unsure if that's valid regex, but you get the point), it's more explicit that way, being very obvious about what characters are included, as well as immediately being clear (to me) about the intention of the character class. "0x20 to 0x7F" triggers the idea of "Printable ASCII" a lot sooner than <space> to <tilde>.