13 ms·
In search of the perfect URL validation regex
- matt2000 11y agoNice to have this comparison in one place, but I think also serves as a good illustration of the limits of regex usefulness. It feels like this would just be better off implemented in the language of your choice instead of an impenetrable string of characters.
- treve 11y agoThis is partially because every single regex in the example is terribly written. Regexes _can_ look good and understandable with the x modifier and lots of spaces and comments. You wouldn't write a normal function in one line with no comments, why do it with regex?
- ethbro 11y agoObligatory "Conway's GoL in one line of APL" http://catpad.net/michael/apl/ http://catpad.net/michael/apl/
- thaumaturgy 11y agoThat can get ugly too. I have a function that takes in a (possibly malformed) URL and returns a key-value array of its parts ... seems straightforward, but it's about 300 lines, and this site includes some tests that I haven't bothered to concoct yet, so mine probably isn't even complete. URIs are one of those things that make you go, "Oh! That should be easy!", and then a week later you're walking around looking for puppies to kick.
- Retra 11y agoThere're some other options for regex syntax: https://github.com/VerbalExpressions/JSVerbalExpressions https://github.com/VerbalExpressions/JSVerbalExpressions
- jakeogh 11y agoI'm using a modified version of this for IRI/URI validation: https://github.com/nisavid/spruce-iri/blob/master/spruce/iri/goose.py https://github.com/nisavid/spruce-iri/blob/master/spruce/iri...
- josephscott 11y agoEyeballing those charts it looks like @scottgonzales scores the best, followed by @cowboy.
- nylar 11y ago@diegoperini seems to be the clear winner with green in each cell.
- function_seven 11y agoLooks like @diegoperini has a perfect score. (Site is wide so if your screen is narrower, you won't see his)
- josephscott 11y agoYou are correct, I missed that one.
- msane 11y agoGreat matrix
- gtrubetskoy 11y agoWhat about an IPv6 URL, e.g. http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.ht...
- recentdarkness 11y agoExactly that is what I was thinking when I checked for the tests and I haven't seen anything in regards to that. If they really want to get the real deal, they would have to support that too. However as others already pointed out, it's better not to use a regex for it, but a proper library for your language which would bail out as soon as it hits something invalid. With the hope that the library for the language does support that already.
- criley2 11y agoWhy should URL's like http://3628126748 http://3628126748 fail? For example, http://1249711460 http://1249711460 resolves just fine in Chrome.
- sombremesa 11y agoAlso appears to resolve in curl.
- eugenekolo2 11y agoWhat is this witchcraft?
- facetube 11y agoIt's a 32-bit IP address in decimal (sorry, base ten) form, IIRC.
- hueving 11y agoUnsigned integer form! /ducks
- rnnr 11y agoYep, I 1st learn this trick in my teens when a dude was using 3564020356 as an address. Certainly a witch nest. :]
- eugenekolo2 11y agoFigured it out also as base 256 ipv4 converted to a number. 74.125.21.100 => 100x1 + 21x256 + 125x256^2 + 74x256^3 => 1249711460
- jetpks 11y agoWhy are these in the `should fail' section? http://www.foo.bar./ http://10.1.1.1 http://10.1.1.254 http://10.1.1.0 http://10.1.1.255 The top one is just a fully qualified domain name, and the rest could be valid host addresses depending on your subnet size.
- kinghajj 11y agoThe first should be allowed, but I can see why the latter are excluded. If the purpose of this validator is to run against user-submitted URLs, then you wouldn't want to allow users to specify ones to which they don't have authority. Imagine that the program takes the user's input, fetches the item, and the stores it somewhere accessible to them. If the program allowed URLs for internal subnets, then a malicious user could attempt to acquire internal assets!
- vacri 11y agoRegarding the IPs, the article's use-case is listed as a 'public URL shortener', and 10.0.0.0/8 is not on the public address range by definition.
- ghayes 11y agoSure, but it seems like you should in those cases just parse the URL and then reject black-listed host names.
- gburt 11y ago
- seba_dos1 11y agoWhat about new TLDs? URLs with them should be also checked. I can see that some regexes contain list of TLDs, which is already disqualifying.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- colinbartlett 11y agoA huge pet peeve of mine are sites that reject the new TLDs in emails or URLs. I use my hotfresh.pizza email address pretty often and you'd be surprised how many sites reject that with "Please enter a valid email address." Infuriating.
- dmd 11y agoYou'd be surprised how many sites reject my domain, 3e.org. Sometimes it's because it begins with a number, sometimes it's because it's too short.
- ikeboy 11y agoI've had Gogo's inflight wifi not let me sign up because my email account at gmail started with a number.
- LukeShu 11y agoTo be fair, an old RFC referenced by an RFC that is referenced by the current thing specifies that a segment of a domain name can't start with a number. Obviously, that isn't true in the real world.
- daegloe 11y agoSame issue with our hi.fi. Facebook's JavaScript SDK wouldn't write cookies to it, instead rejecting it as too short and invalid. Had to open a ticket and the oversight was corrected, with lots of embarrassment and apologizing.
- k_sze 11y agoAnd now you have 404 problems. In all seriousness, this is why I don't like the IETF's documents. They write in a verbose way and then don't even provide a reference implementation, in this case, a reference regex that would have done away with lots of dispute and ambiguity. It is my opinion that, in practice, your specification is inherently broken if you can't provide a reference implementation. (No, BNF doesn't count as reference "implementation". Who uses BNF in their programs to validate strings anyway?)
- SFjulie1 11y agoThere is more than grammar in URL: there is politics like gov.uk is a TLD. And that is why IETF RFC are so verbose. Because there is politics in them. That is why you can't provide regexps. url because of politics are not context free. https://www.cs.rochester.edu/~nelson/courses/csc_173/grammars/cfg.html https://www.cs.rochester.edu/~nelson/courses/csc_173/grammar... So a URL because of politics is a context full grammar. And regexp cannot parse anything else than context free stuff. To be honest I doubt anything useful is context free. I doubt, therefore, that anything useful can be parsed with regexp... Except float, integer, and other basic types that are useful to build a context full grammar... But for this RFC should separate standards in context free stuffs (rules/context free stuff) and config file for the political/commercial stuffs (context that changes meanings of atoms being parsed for illogical reasons). The problem is politics is fucking hard to normalize, we have no BSML yet.
- pekk 11y agoA minor clarification for those who might be learning from HN: regular expressions aren't equivalent to context-free grammars, they are even more limited than context-free grammars. Things which involve arbitrary-depth nesting or recursion, like HTML, can't be parsed by regular expressions, but that is exactly what context-free grammars are for. Where regular expressions correspond to finite state automata (essentially directed graphs where nodes are states and edges are input-driven transitions between states), context-free grammars correspond to pushdown automata (with a stack, this is the recursion secret sauce). Lots of interesting things are context-free. For example, any good programming language grammar should be context-free, to allow sane and efficient parsing. I welcome further corrections.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- Ov3rload 11y agoWhy would you ever use a regex to do this, apart from experimenting? This would be much easier to do with a context free grammar or any decent parsing library.
- ipozgaj 11y agoA lot of these regexes miss the whole point of validation. The key problem that needs to be solved here is to find if a URL is syntactically correct. A lot of them focus on the semantics as well. Thus hardcoding the TLDs is not needed at all, so http://microsoft.com http://microsoft.com and http://microsoft.foobar http://microsoft.foobar should both be valid (I could have the latter in my /etc/hosts). Also, that's what the DNS is used for.
- OliverJones 11y agoTerrific. Sr. Perini's regex has served me well. That being said, if you solve a problem with a list of thirteen regexs, you then have fourteen problems.
- noondip 11y agohttp://./ http://./ is a perfectly valid and very short URL, even though it is highly unlikely anyone (InterNIC? ICANN? Verisign?) would issue an address record for @. On a related note, has anyone seen ccTLDs do something like this (e.g., http://io./ http://io./, https://co.uk./foo https://co.uk./foo)?
- noinsight 11y ago> On a related note, has anyone seen ccTLDs do something like this (e.g., http://io./ http://io./, https://co.uk./foo https://co.uk./foo)? Yes, I wondered about this myself, http://dk http://dk. redirects to www.dk-hostmaster.dk. It would be interesting to receive mail from root@com or whatever :)
- nitrogen 11y agoThe Vatican advertises MX records on va.: $ dig va MX ... ;; ANSWER SECTION: va. 3599 IN MX 10 mx12.vatican.va. va. 3599 IN MX 10 mx11.vatican.va. va. 3599 IN MX 100 raphaelmx3.posta.va.
- noondip 11y agoCool! Anyone want to try emailing pope@va and seeing what comes of it?
- javan 11y agoHere's my easy (but not totally passing) version using the DOM: https://gist.github.com/javan/6aaebfeb5fe415498028 https://gist.github.com/javan/6aaebfeb5fe415498028
- codezero 11y agoMaybe gwern can build a neural net for this too. :)
- bazzargh 11y agoThe one marked as winning, by @diegoperini, has problems. The way it defines passwords is wrong, so you can trick it into accepting almost anything by putting the domain somewhere else: re_weburl.test("http://127.0.0.1/") => false re_weburl.test("http://127.0.0.1/@example.com") => true re_weburl.test("http://999.999.999.999.999/@example.com") => true oops. I disagree that it should be rejecting rfc1918 addresses anyway, because this makes it less useful in an intranet context, where you want those to work. There's also an apples-to-oranges comparison going on here. The Gruber pattern is not for validation, but for detecting url-like-things in text, which is why it excludes a whole bunch of punctuation chars from appearing at the end - when I say 'google.com.' in text, I mean 'google.com'. Edited to add: I misremembered suggesting trailing punctuation exclusion to Gruber, what we discussed was xxx.xxx/xxx as an alternate pattern, catching protocol-less shortened urls in tweets.
- deleted 11y ago[deleted]
- matrix 11y agoWhile I admire the effort that people put into this (and the similar effort of email address validation), what I really want to see is a comparison based on "good enough" validation -vs- performance. I think a reasonably low rate false-positives is a reasonable trade-off for fast validation.
- billyhoffman 11y agoPlease. Please Please Please. With Sugar on Top. Don't do this with a regex. I've written a lot of web facing software that accepts URLs from the untrusted masses and ultimately makes requests to them if they are "valid." The lesson I've learned is simple. Regex's are terrible for this task because there are a ton of things you check and lots of normalization you need to do. Instead, do this as a function I've evolved mine over the years, and my use case is semi-specific: Given a string, validate it as a fully qualified HTTP/HTTPS URL that doesn't have credentials and isn't trying to point my software toward the internal network or localhost. It looks like this: - Use system/framework library to create Uri object from source string. All your checks will be consulting this object's properties, not looking at the source string - Is scheme HTTP/HTTPS? If no, stop - Did they supply user:pass@ in URL? If so, stop, and yell at them for putting usernames and passwords into a random site on the Internet. - If hostname is an IP address, normalize it to dotted decimal quad IPv4 or IPv6 (no octal obfuscation for you!), and test against private IP space ranges or loopback. If private or loopback, stop - If hostname is an actual hostname, normalize it + de-puny code it, and check for localhost aliases. If local, stop (you can also do a DNS lookup and make sure you someone isn't trying to return private/local IPs to bypass your checks) At this point, you have a syntactically valid, fully qualified URL pointing to a public facing web property accessed via HTTP or HTTPS. You don't have to worry about TLDS or the like. At this point, you can do additional DNS checks, check the domain against lists of bad actors, whatever else you want to do. You can try and be smart and do things like, "if supplied URL wasn't fully qualified, prepend http:// http:// and try validation again" to avoid user error. Pretty flexible. This is more rigorous than a simple regex and way way way easier for another developer to read and understand what is going on.
- elithrar 11y ago> - Did they supply user:pass@ in URL? If so, stop, and yell at them for putting usernames and passwords into a random site on the Internet. FWIW, browsers don't send the user:pass in the URL - they automatically marshal it into the Authorization: Basic header. Obviously sending these over HTTP is still dumb, but they're not (typically) logged in plain-text in browser history/server request logs.
- aruggirello 11y agoI just don't understand why so many people like to reinvent the wheel all the time. In PHP, there's filter_var() family of functions with plenty of filter types, like FILTER_VALIDATE_URL, or you could try to use parse_url() if you need to add further constrains to validated URLs - like forbidding localhost, etc. IMHO complex regular expressions should be avoided as they make debugging a PITA and are usually a performance bottleneck, too.
- hpaavola 11y agoIn many cases you don't want "http://foo.bar?q=Spaces http://foo.bar?q=Spaces should be encoded" to pass. If for example you want to turn URLs in comments to links, then space should just end the URL right there. Otherwise you end up making whole paragraphs as links.