5 ms·

It seems nobody really understands how to handle non-ascii URLs. To get it right for Scrapy (i.e. do same as browsers) we checked how several browsers behave; t

by kmike84 10y ago

It seems nobody really understands how to handle non-ascii URLs. To get it right for Scrapy (i.e. do same as browsers) we checked how several browsers behave; the results was consistent but surprising, if not crazy. Before sending an URL to a server (e.g. when user clicks a link) browser does the following: 1. Url is parsed into components. 2. Domain name is encoded using punycode. 3. URL path is encoded to UTF-8 and then percent-escaped. 4. Query string and fragment are encoded using page encoding and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding. 5. The result is joined back to a single byte string and sent to a server. So this means parts of a single URL are encoded using 3 different encodings, and one needs to know the encoding of a page URL was extracted from to send it correctly (i.e. to send it the way browsers do). I haven't seen this algorithm stated explicitly anywhere, but this is how all browsers work. It doesn't make any sense.

tfm 10y ago
That's some pretty useful documentation there. I hope that it at least forms part of a submission to the WHATWG folks.
oneeyedpigeon 10y ago
> For URLs entered to address bar it seems that UTF-8 is used, not page encoding. Maybe I'm misunderstanding, but how could a browser use page-encoding before the page has been fetched?
- kmike84 10y ago
  It uses encoding of a page URL is extracted from, not of a page URL leads to.
- wolfgke 10y ago
  That's how: https://html.spec.whatwg.org/multipage/syntax.html#determining-the-character-encoding https://html.spec.whatwg.org/multipage/syntax.html#determini...
  - ingenter 10y ago
    And how would you deal with an URL to the HTML itself?
    - kmike84 10y ago
      There are 3 ways to get an URL: a) URL is entered to the address bar. Default encoding (it seems UTF-8) is used for query and fragment. b) User clicks on a link. Page URL is used; page encoding is already known. c) Redirect to a new URL happens. I can't say out of my head what happens in this case.
      - jrochkind1 10y ago
        This is just a recipe for massive confusion though. You're saying if I click on a link with an IRI in a non-UTF8 html page, it means one thing. But if I copy and paste that link into a text file, and later paste it into a browser, it'll mean something else entirely? Or if I save it to a bookmark using some service?
        kmike84 10y ago
        Yeah, it is a recipe for confusion. Browsers use a workaround: when you copy an URL of a link (using "Copy Link Address" in Chrome or "Copy Link Location" in Firefox) it copies escaped URL, not original URL.
bdarnell 10y ago
In theory, this is specified in section 3.1 of https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt (including the "MAY" section beginning "Systems accepting IRIs MAY convert the ireg-name component of an IRI as follows"). That covers the distinction between the domain and the rest of the URL. The use of the page encoding is part of the browser behavior, not any RFC. It has been retroactively specified in https://url.spec.whatwg.org/#url-parsing https://url.spec.whatwg.org/#url-parsing
- kmike84 10y ago
  Thanks for the references!
pygy_ 10y ago
However, the JS gettters are all over the place... Given this: a = document.createElement('a') a.href = "/ö/o?ö/o#ö/o" console.log(a.hash, a.pathname, a.search) we get: Browser .hash .pathname .search Chrome #ö/o /%C3%B6/o ?%C3%B6/o Firefox #%C3%B6/o /%C3%B6/o ?%C3%B6/o Safari #%C3%B6/o /%C3%B6/o ?%C3%B6/o Edge #ö/o /ö/o ?ö/o IE11 #ö/o ö/o (ouch) ?ö/o Likewise for domains, some UAs return them Punycoded (but don't provide a decoder), some in Unicode, and PhantomJS passes it percent encoded... It's a big mess. I discovered this while working on a SPA router that can handle routes in either `pathname`, `search` or `hash` mode... The routes are defined as the keys of a JS object, in Unicode in the source, and we have to juggle back and forth to match what `document.location.*` returns.
- kmike84 10y ago
  It looks like Developer tools also use these JS utilities, so you can't trust Network tab when checking what URL is sent to a remote server; to check what's being sent one have to either inspect network traffic or check it server side.
thaumasiotes 10y ago
> Query string and fragment are encoded using page encoding and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding. Well, for links that appear in a page to be defined using page encoding doesn't sound so shocking. That doesn't explain why the path would use UTF-8 regardless of page encoding, though. Are you sure URLs directly entered in the address bar use UTF-8 for query strings and fragments rather than system encoding? If my OS is set to use GB2312, how do URL fragments that I type into the browser get encoded?
- kmike84 10y ago
  A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding. This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do when page encoding is unknown; unlike encoding of URLs extracted from HTML this UTF-8 default doesn't lead to incompatible behavior.
bzbarsky 10y ago
The not making sense bit is a consequence of several waves of changes, some of them not really involving standards. The first thing to be added here, I expect, was the handling of the query string. This was needed to allow non-ASCII things to be submitted via forms; the need for this arose before there was any support for non-ASCII domain names, and likely before there was any real need for non-ascii paths. The issue with forms as they were initially created is that they would submit to some server and the server would then process the query string. In doing so it would do whatever it did for non-ASCII stuff; there was no standard. In practice, it would percent-unescape and then treat the bytes as being in whatever encoding the server developer happened to default to. Typically this was the encoding the web page was in as well. Yes, this is totally busted if your form has a name field and the name being input is not representable in ISO-8859-1 or whatever you authored your page in, but this was an incredibly common way to handle non-ASCII in the 90s. This all predates my involvement in browsers, so I'm not sure which is the chicken and which is the egg here, but the upshot was that browsers started sending the query string in the page encoding (with various hacks that were not interoperable for a while in cases when the text was not representable in that encoding) and servers started depending on this behavior. Then the accept-charset attribute got added to <form> to allow overriding the default behavior for people who wanted something else. The behavior of form submission via the query string in terms of encodings is specified at https://html.spec.whatwg.org/multipage/forms.html#picking-an-encoding-for-the-form https://html.spec.whatwg.org/multipage/forms.html#picking-an... and https://url.spec.whatwg.org/#concept-urlencoded-serializer https://url.spec.whatwg.org/#concept-urlencoded-serializer and https://html.spec.whatwg.org/multipage/forms.html#url-encoded-form-data https://html.spec.whatwg.org/multipage/forms.html#url-encode... which bridges them. I think the next step was url paths; for those browsers were inconsistent for a bit about whether UTF-8 was used or whether the page encoding was used, but eventually all aligned on UTF-8, and this got standardized in the HTML spec as well, I'm fairly certain. And for hostnames, punycode encoding was more or less a must for doing DNS resolution. And at that point that ended up getting sent on the wire as well. https://tools.ietf.org/html/rfc2616#section-14.23 https://tools.ietf.org/html/rfc2616#section-14.23 defines the Host header, which is presumably what we're talking about here, since that's the only way the hostname is sent to the server (note that this is NOT part of the same byte string as the path and query). Following the breadcrumbs for that through https://tools.ietf.org/html/rfc2396 https://tools.ietf.org/html/rfc2396 it all talks about this header sending the DNS name, which is why I assume browsers settled on sending punycode here. %-encoding in hostnames wasn't really a thing for a while, I think; e.g. Firefox didn't even support it until about 6 months ago.

by kmike84 10y ago

tfm 10y ago

That's some pretty useful documentation there. I hope that it at least forms part of a submission to the WHATWG folks.

oneeyedpigeon 10y ago

> For URLs entered to address bar it seems that UTF-8 is used, not page encoding. Maybe I'm misunderstanding, but how could a browser use page-encoding before the page has been fetched?

kmike84 10y ago
It uses encoding of a page URL is extracted from, not of a page URL leads to.
wolfgke 10y ago
That's how: https://html.spec.whatwg.org/multipage/syntax.html#determining-the-character-encoding https://html.spec.whatwg.org/multipage/syntax.html#determini...
- ingenter 10y ago
  And how would you deal with an URL to the HTML itself?
  - kmike84 10y ago
    There are 3 ways to get an URL: a) URL is entered to the address bar. Default encoding (it seems UTF-8) is used for query and fragment. b) User clicks on a link. Page URL is used; page encoding is already known. c) Redirect to a new URL happens. I can't say out of my head what happens in this case.
    - jrochkind1 10y ago
      This is just a recipe for massive confusion though. You're saying if I click on a link with an IRI in a non-UTF8 html page, it means one thing. But if I copy and paste that link into a text file, and later paste it into a browser, it'll mean something else entirely? Or if I save it to a bookmark using some service?
      - kmike84 10y ago
        Yeah, it is a recipe for confusion. Browsers use a workaround: when you copy an URL of a link (using "Copy Link Address" in Chrome or "Copy Link Location" in Firefox) it copies escaped URL, not original URL.

bdarnell 10y ago

In theory, this is specified in section 3.1 of https://www.ietf.org/rfc/rfc3987.txt https://www.ietf.org/rfc/rfc3987.txt (including the "MAY" section beginning "Systems accepting IRIs MAY convert the ireg-name component of an IRI as follows"). That covers the distinction between the domain and the rest of the URL. The use of the page encoding is part of the browser behavior, not any RFC. It has been retroactively specified in https://url.spec.whatwg.org/#url-parsing https://url.spec.whatwg.org/#url-parsing

kmike84 10y ago
Thanks for the references!

pygy_ 10y ago

However, the JS gettters are all over the place... Given this: a = document.createElement('a') a.href = "/ö/o?ö/o#ö/o" console.log(a.hash, a.pathname, a.search) we get: Browser .hash .pathname .search Chrome #ö/o /%C3%B6/o ?%C3%B6/o Firefox #%C3%B6/o /%C3%B6/o ?%C3%B6/o Safari #%C3%B6/o /%C3%B6/o ?%C3%B6/o Edge #ö/o /ö/o ?ö/o IE11 #ö/o ö/o (ouch) ?ö/o Likewise for domains, some UAs return them Punycoded (but don't provide a decoder), some in Unicode, and PhantomJS passes it percent encoded... It's a big mess. I discovered this while working on a SPA router that can handle routes in either `pathname`, `search` or `hash` mode... The routes are defined as the keys of a JS object, in Unicode in the source, and we have to juggle back and forth to match what `document.location.*` returns.

kmike84 10y ago
It looks like Developer tools also use these JS utilities, so you can't trust Network tab when checking what URL is sent to a remote server; to check what's being sent one have to either inspect network traffic or check it server side.

thaumasiotes 10y ago

> Query string and fragment are encoded using page encoding and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding. Well, for links that appear in a page to be defined using page encoding doesn't sound so shocking. That doesn't explain why the path would use UTF-8 regardless of page encoding, though. Are you sure URLs directly entered in the address bar use UTF-8 for query strings and fragments rather than system encoding? If my OS is set to use GB2312, how do URL fragments that I type into the browser get encoded?

kmike84 10y ago
A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding. This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do when page encoding is unknown; unlike encoding of URLs extracted from HTML this UTF-8 default doesn't lead to incompatible behavior.

bzbarsky 10y ago

The not making sense bit is a consequence of several waves of changes, some of them not really involving standards. The first thing to be added here, I expect, was the handling of the query string. This was needed to allow non-ASCII things to be submitted via forms; the need for this arose before there was any support for non-ASCII domain names, and likely before there was any real need for non-ascii paths. The issue with forms as they were initially created is that they would submit to some server and the server would then process the query string. In doing so it would do whatever it did for non-ASCII stuff; there was no standard. In practice, it would percent-unescape and then treat the bytes as being in whatever encoding the server developer happened to default to. Typically this was the encoding the web page was in as well. Yes, this is totally busted if your form has a name field and the name being input is not representable in ISO-8859-1 or whatever you authored your page in, but this was an incredibly common way to handle non-ASCII in the 90s. This all predates my involvement in browsers, so I'm not sure which is the chicken and which is the egg here, but the upshot was that browsers started sending the query string in the page encoding (with various hacks that were not interoperable for a while in cases when the text was not representable in that encoding) and servers started depending on this behavior. Then the accept-charset attribute got added to <form> to allow overriding the default behavior for people who wanted something else. The behavior of form submission via the query string in terms of encodings is specified at https://html.spec.whatwg.org/multipage/forms.html#picking-an-encoding-for-the-form https://html.spec.whatwg.org/multipage/forms.html#picking-an... and https://url.spec.whatwg.org/#concept-urlencoded-serializer https://url.spec.whatwg.org/#concept-urlencoded-serializer and https://html.spec.whatwg.org/multipage/forms.html#url-encoded-form-data https://html.spec.whatwg.org/multipage/forms.html#url-encode... which bridges them. I think the next step was url paths; for those browsers were inconsistent for a bit about whether UTF-8 was used or whether the page encoding was used, but eventually all aligned on UTF-8, and this got standardized in the HTML spec as well, I'm fairly certain. And for hostnames, punycode encoding was more or less a must for doing DNS resolution. And at that point that ended up getting sent on the wire as well. https://tools.ietf.org/html/rfc2616#section-14.23 https://tools.ietf.org/html/rfc2616#section-14.23 defines the Host header, which is presumably what we're talking about here, since that's the only way the hostname is sent to the server (note that this is NOT part of the same byte string as the path and query). Following the breadcrumbs for that through https://tools.ietf.org/html/rfc2396 https://tools.ietf.org/html/rfc2396 it all talks about this header sending the DNS name, which is why I assume browsers settled on sending punycode here. %-encoding in hostnames wasn't really a thing for a while, I think; e.g. Firefox didn't even support it until about 6 months ago.