3 ms·

Unfortunately, simple URL parsing breaks on so many things. There is a reason on why every URL parsing library is at least a few thousand LOCs. One common way

by f311a 2mo ago

Unfortunately, simple URL parsing breaks on so many things. There is a reason on why every URL parsing library is at least a few thousand LOCs. One common way to test it is just to pass ipv6 url: http://[f021:d981:b487:e57d:193e:550e::]/

meindnoch 2mo ago
Is that so? RFC 3986 Appendix B [1] "Parsing a URI Reference with a Regular Expression": The following line is the regular expression for breaking-down a well-formed URI reference into its components. ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? scheme = $2 authority = $4 path = $5 query = $7 fragment = $9 Let's test your URI with this regex, shall we? [2] $2 (scheme) = http $4 (authority) = [f021:d981:b487:e57d:193e:550e::] $5 (path) = / Seems correct to me. [1] https://datatracker.ietf.org/doc/html/rfc3986#appendix-B https://datatracker.ietf.org/doc/html/rfc3986#appendix-B [2] https://regexr.com/8nqop https://regexr.com/8nqop
- quuxplusone 2mo ago
  Hm, but I think he's right. The problem comes when you try to break down the authority portion into host and port; TFA's parser treats the first colon as introducing the port, which is wrong. https://github.com/bkaradzic/bx/blob/0b001f5f36579e8aea07efa5af139ca18dad9505/src/url.cpp#L48-L123 https://github.com/bkaradzic/bx/blob/0b001f5f36579e8aea07efa...
  - meindnoch 2mo ago
    >The problem comes when you try to break down the authority portion into host and port That's a problem orthogonal to URI parsing. You parse the URI with the RFC 3986 regex, which gives you the components: scheme, authority, path, query, fragment. You're then free to parse any of the components according to your own bespoke rules, e.g. the query string often follows the key=value&key=value&... pattern.
- f311a 2mo ago
  Regexes are pretty slow, though.
  - meindnoch 2mo ago
    cc @burntsushi
- afiori 2mo ago
  > well-formed URI A parser that assumes the input to be already valid is usually not enough for most applications according to that regex this is a valid url __..__..%%%zz..
  - meindnoch 2mo ago
    >according to that regex this is a valid url >__..__..%%%zz.. Correctly. It is a valid relative URI, whose path is "__..__..%%%zz..".
    - afiori 2mo ago
      % characters must be escaped in URIs at the very least, and _ is not valid in domains also this one is valid too for the regex %_%://%_%.%_%/%_%?%_%#%_%
      - afiori 2mo ago
        courtesy of chatgpt there is also this abomination ht tp://[\x00::gg]\t/foo\nbar/%/%QZ/%F?q=<\"|{}>##
r3d 2mo ago
Yeah it's complicated, and that's the thing about parsing anything, the more complicated and unpredictable the input and the harder it is to parse. Does it need to be human readable, does it need to work across all platforms. Does the it need to be secure. These things change everything. Speed, reliability, security pick one. Your point about being compliant with the real spec is the difference between a 20 line scannf and and a 1000 line function. Yeah. Ha.

by f311a 2mo ago

meindnoch 2mo ago

Is that so? RFC 3986 Appendix B [1] "Parsing a URI Reference with a Regular Expression": The following line is the regular expression for breaking-down a well-formed URI reference into its components. ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? scheme = $2 authority = $4 path = $5 query = $7 fragment = $9 Let's test your URI with this regex, shall we? [2] $2 (scheme) = http $4 (authority) = [f021:d981:b487:e57d:193e:550e::] $5 (path) = / Seems correct to me. [1] https://datatracker.ietf.org/doc/html/rfc3986#appendix-B https://datatracker.ietf.org/doc/html/rfc3986#appendix-B [2] https://regexr.com/8nqop https://regexr.com/8nqop

quuxplusone 2mo ago
Hm, but I think he's right. The problem comes when you try to break down the authority portion into host and port; TFA's parser treats the first colon as introducing the port, which is wrong. https://github.com/bkaradzic/bx/blob/0b001f5f36579e8aea07efa5af139ca18dad9505/src/url.cpp#L48-L123 https://github.com/bkaradzic/bx/blob/0b001f5f36579e8aea07efa...
- meindnoch 2mo ago
  >The problem comes when you try to break down the authority portion into host and port That's a problem orthogonal to URI parsing. You parse the URI with the RFC 3986 regex, which gives you the components: scheme, authority, path, query, fragment. You're then free to parse any of the components according to your own bespoke rules, e.g. the query string often follows the key=value&key=value&... pattern.
f311a 2mo ago
Regexes are pretty slow, though.
- meindnoch 2mo ago
  cc @burntsushi
afiori 2mo ago
> well-formed URI A parser that assumes the input to be already valid is usually not enough for most applications according to that regex this is a valid url __..__..%%%zz..
- meindnoch 2mo ago
  >according to that regex this is a valid url >__..__..%%%zz.. Correctly. It is a valid relative URI, whose path is "__..__..%%%zz..".
  - afiori 2mo ago
    % characters must be escaped in URIs at the very least, and _ is not valid in domains also this one is valid too for the regex %_%://%_%.%_%/%_%?%_%#%_%
    - afiori 2mo ago
      courtesy of chatgpt there is also this abomination ht tp://[\x00::gg]\t/foo\nbar/%/%QZ/%F?q=<\"|{}>##

r3d 2mo ago

Yeah it's complicated, and that's the thing about parsing anything, the more complicated and unpredictable the input and the harder it is to parse. Does it need to be human readable, does it need to work across all platforms. Does the it need to be secure. These things change everything. Speed, reliability, security pick one. Your point about being compliant with the real spec is the difference between a 20 line scannf and and a 1000 line function. Yeah. Ha.