4 ms·

Please. Please Please Please. With Sugar on Top. Don't do this with a regex. I've written a lot of web facing software that accepts URLs from the untrusted mas

Please. Please Please Please. With Sugar on Top. Don't do this with a regex. I've written a lot of web facing software that accepts URLs from the untrusted masses and ultimately makes requests to them if they are "valid." The lesson I've learned is simple. Regex's are terrible for this task because there are a ton of things you check and lots of normalization you need to do. Instead, do this as a function I've evolved mine over the years, and my use case is semi-specific: Given a string, validate it as a fully qualified HTTP/HTTPS URL that doesn't have credentials and isn't trying to point my software toward the internal network or localhost. It looks like this: - Use system/framework library to create Uri object from source string. All your checks will be consulting this object's properties, not looking at the source string - Is scheme HTTP/HTTPS? If no, stop - Did they supply user:pass@ in URL? If so, stop, and yell at them for putting usernames and passwords into a random site on the Internet. - If hostname is an IP address, normalize it to dotted decimal quad IPv4 or IPv6 (no octal obfuscation for you!), and test against private IP space ranges or loopback. If private or loopback, stop - If hostname is an actual hostname, normalize it + de-puny code it, and check for localhost aliases. If local, stop (you can also do a DNS lookup and make sure you someone isn't trying to return private/local IPs to bypass your checks) At this point, you have a syntactically valid, fully qualified URL pointing to a public facing web property accessed via HTTP or HTTPS. You don't have to worry about TLDS or the like. At this point, you can do additional DNS checks, check the domain against lists of bad actors, whatever else you want to do. You can try and be smart and do things like, "if supplied URL wasn't fully qualified, prepend http:// http:// and try validation again" to avoid user error. Pretty flexible. This is more rigorous than a simple regex and way way way easier for another developer to read and understand what is going on.

elithrar 11y ago
> - Did they supply user:pass@ in URL? If so, stop, and yell at them for putting usernames and passwords into a random site on the Internet. FWIW, browsers don't send the user:pass in the URL - they automatically marshal it into the Authorization: Basic header. Obviously sending these over HTTP is still dumb, but they're not (typically) logged in plain-text in browser history/server request logs.
- billyhoffman 11y ago
  Ahh. Very true. Let me be a little more clear. I work on public web apps where there is literally an HTML text input for someone to submit a URL that we should audit. I'm not talking about people typing URLs directly into a browser. At least once a month I get someone giving me a URL with embedded credentials to a dev or staging environment for an Alexa top 50K site. Things like https://qa:tester@dev.major-ecomm-site.com/blah https://qa:tester@dev.major-ecomm-site.com/blah. It's pretty terrible, but makes for an interesting sales call :-)
- moloch 11y ago
  It can affect what domain you end up at though, for example: http://foo:bar.com\@example.com http://foo:bar.com\@example.com Chrome/Webkit browsers will go to `bar.com` but Firefox will end up at `example.com` for the same URL! There's a fantastic book called "The Tangled Web" that has lots of examples of these pitfalls.
agwa 11y ago
I'm extremely impressed you thought of checking for private IP addresses - everyone seems to forget that. But as you've described it, there's a time-of-check/time-of-use vulnerability: an attacker could set a really low TTL on the A record and swap it out with a private address between your check and the time you actually hit the URL. You really have to hook into the HTTP client for that check (in Perl, I recommend LWPx::ParanoidAgent; if using libcurl, you can use a CURLOPT_OPENSOCKETFUNCTION callback).
pjtr 11y ago
Agreed. It's "funny" how common 1000+ character uncommented oneliner regular expressions are, while in any other programming language even things like `if (flag) return;` would get rejected for being dangerous, unreadable, uncommented, brace style violations and possibly even for being too long if indentation and length of the flag name make this exceed just 80 characters. Is this because people think of a regex as an atomic blackbox instead of a program or function that can be read and modified? For example when a regex is incorrect, they don't say "This regex is incorrect. How can it be corrected?" like they would for a program or function, but "This is the incorrect regex. Which one is the correct regex." Free-spacing mode (where you can add whitespace, newlines and comments to a regex) would at least help a little. (Although even then I think a function with early returns is often more appropriate. Possibly using very small regular expressions for some of the individual checks.) But I rarely even see free-spacing modes mentioned. Maybe because most programmers using regex actually prefer an atomic blackbox?
- charriu 11y ago
  Most programmers I've met are not in fact regex "literate". They may be able to write one, but understanding what one does, or even modifying it? Well... So yeah, black box.
pluma 11y ago
Using a regex to validate a URL is only marginally less stupid than using a regex to validate an e-mail address. Validating URLs is also something you shouldn't have to implement yourself.
- nalllar 11y ago
  Validating email addresses with a regex is fine if you follow the HTML5 spec for email addresses: http://www.w3.org/TR/html5/forms.html#valid-e-mail-address http://www.w3.org/TR/html5/forms.html#valid-e-mail-address > This requirement is a willful violation of RFC 5322, which defines a syntax for e-mail addresses that is simultaneously too strict (before the "@" character), too vague (after the "@" character), and too lax (allowing comments, whitespace characters, and quoted strings in manners unfamiliar to most users) to be of practical use here.

elithrar 11y ago

> - Did they supply user:pass@ in URL? If so, stop, and yell at them for putting usernames and passwords into a random site on the Internet. FWIW, browsers don't send the user:pass in the URL - they automatically marshal it into the Authorization: Basic header. Obviously sending these over HTTP is still dumb, but they're not (typically) logged in plain-text in browser history/server request logs.

billyhoffman 11y ago
Ahh. Very true. Let me be a little more clear. I work on public web apps where there is literally an HTML text input for someone to submit a URL that we should audit. I'm not talking about people typing URLs directly into a browser. At least once a month I get someone giving me a URL with embedded credentials to a dev or staging environment for an Alexa top 50K site. Things like https://qa:tester@dev.major-ecomm-site.com/blah https://qa:tester@dev.major-ecomm-site.com/blah. It's pretty terrible, but makes for an interesting sales call :-)
moloch 11y ago
It can affect what domain you end up at though, for example: http://foo:bar.com\@example.com http://foo:bar.com\@example.com Chrome/Webkit browsers will go to `bar.com` but Firefox will end up at `example.com` for the same URL! There's a fantastic book called "The Tangled Web" that has lots of examples of these pitfalls.

agwa 11y ago

I'm extremely impressed you thought of checking for private IP addresses - everyone seems to forget that. But as you've described it, there's a time-of-check/time-of-use vulnerability: an attacker could set a really low TTL on the A record and swap it out with a private address between your check and the time you actually hit the URL. You really have to hook into the HTTP client for that check (in Perl, I recommend LWPx::ParanoidAgent; if using libcurl, you can use a CURLOPT_OPENSOCKETFUNCTION callback).

pjtr 11y ago

Agreed. It's "funny" how common 1000+ character uncommented oneliner regular expressions are, while in any other programming language even things like `if (flag) return;` would get rejected for being dangerous, unreadable, uncommented, brace style violations and possibly even for being too long if indentation and length of the flag name make this exceed just 80 characters. Is this because people think of a regex as an atomic blackbox instead of a program or function that can be read and modified? For example when a regex is incorrect, they don't say "This regex is incorrect. How can it be corrected?" like they would for a program or function, but "This is the incorrect regex. Which one is the correct regex." Free-spacing mode (where you can add whitespace, newlines and comments to a regex) would at least help a little. (Although even then I think a function with early returns is often more appropriate. Possibly using very small regular expressions for some of the individual checks.) But I rarely even see free-spacing modes mentioned. Maybe because most programmers using regex actually prefer an atomic blackbox?

charriu 11y ago
Most programmers I've met are not in fact regex "literate". They may be able to write one, but understanding what one does, or even modifying it? Well... So yeah, black box.

pluma 11y ago

Using a regex to validate a URL is only marginally less stupid than using a regex to validate an e-mail address. Validating URLs is also something you shouldn't have to implement yourself.

nalllar 11y ago
Validating email addresses with a regex is fine if you follow the HTML5 spec for email addresses: http://www.w3.org/TR/html5/forms.html#valid-e-mail-address http://www.w3.org/TR/html5/forms.html#valid-e-mail-address > This requirement is a willful violation of RFC 5322, which defines a syntax for e-mail addresses that is simultaneously too strict (before the "@" character), too vague (after the "@" character), and too lax (allowing comments, whitespace characters, and quoted strings in manners unfamiliar to most users) to be of practical use here.