3 ms·

In this case, he initially presumed that the problem would be with Flask, Bottle, and Tornado still honoring a semi-colon as a separator in a uri. You're not s

by tyingq 5y ago

In this case, he initially presumed that the problem would be with Flask, Bottle, and Tornado still honoring a semi-colon as a separator in a uri. You're not supposed to, according to 2014 W3C guidance: "Let strings be the result of strictly splitting the string payload on U+0026 AMPERSAND characters" From: https://www.w3.org/TR/2014/REC-html5-20141028/forms.html#url-encoded-form-data https://www.w3.org/TR/2014/REC-html5-20141028/forms.html#url... They removed the old 1999 guidance that said "in particular, CGI implementors support the use of ";" in place of "&" to save authors the trouble of escaping "&" characters in this manner." As it turns out, the issue was in urllib.parse, a library that ships with python.

charcircuit 5y ago
This seems like a breaking change that can break links throughout the web. It seems better for me to keep the links working and fix the proxies then to break links and change python.
- tyingq 5y ago
  I'm not sure why you keep referencing proxies. Lots of different software uses urllib, so it would affect web servers too...many of them without fronting proxies. Some client side software projects use it too, like scrapers, for example.
  - charcircuit 5y ago
    >I'm not sure why you keep referencing proxies That's what I thought the article was about.
    - tyingq 5y ago
      Ah, okay. It is about web cache poisoning, but there are many web caches that would use urllib.parse without a proxy being involved. Like an app directly counting on urllib.parse to generate a key for a redis cache. And the base problem happens to have broader implications than just cache poisoning. It could cause, for example, plain old data corruption for url parameter values.
jasonhansel 5y ago
From that spec: > If any of the name-value pairs in pairs have a name component consisting of the string "_charset_" encoded in US-ASCII, and the value component of the first such pair, when decoded as US-ASCII, is the name of a supported character encoding, then let encoding be that character encoding (replacing the default passed to the algorithm). Ouch. I hope nobody actually implements that.
- tyingq 5y ago
  I couldn't find support for that in urllib or python requests. But googling a bit does show it's supported in a lot of software. For example: https://bugzilla.mozilla.org/show_bug.cgi?id=18643 https://bugzilla.mozilla.org/show_bug.cgi?id=18643
the_mitsuhiko 5y ago
> In this case, he initially presumed that the problem would be with Flask, Bottle, and Tornado still honoring a semi-colon as a separator in a uri Flask (or rather Werkzeug what it's based on) removed implicit semicolon support for URL 13 years ago: https://github.com/pallets/werkzeug/commit/0ea28bbc6f5f05eefb0e648b52e6400be4b11e43 https://github.com/pallets/werkzeug/commit/0ea28bbc6f5f05eef...
- tyingq 5y ago
  Yeah, it appears Flask doesn't use urllib either. I think he was just outlining his approach though. That he started with popular frameworks, assuming any issues would be there, but ended up in urllib. Bottle does use parts of urllib but appears to do it's own splitting on ampersands: https://github.com/bottlepy/bottle/blob/d1413a81ead7a6f130a06e21a17ca98f4ea30df6/bottle.py#L3053 https://github.com/bottlepy/bottle/blob/d1413a81ead7a6f130a0...

by tyingq 5y ago

charcircuit 5y ago

This seems like a breaking change that can break links throughout the web. It seems better for me to keep the links working and fix the proxies then to break links and change python.

tyingq 5y ago
I'm not sure why you keep referencing proxies. Lots of different software uses urllib, so it would affect web servers too...many of them without fronting proxies. Some client side software projects use it too, like scrapers, for example.
- charcircuit 5y ago
  >I'm not sure why you keep referencing proxies That's what I thought the article was about.
  - tyingq 5y ago
    Ah, okay. It is about web cache poisoning, but there are many web caches that would use urllib.parse without a proxy being involved. Like an app directly counting on urllib.parse to generate a key for a redis cache. And the base problem happens to have broader implications than just cache poisoning. It could cause, for example, plain old data corruption for url parameter values.

jasonhansel 5y ago

From that spec: > If any of the name-value pairs in pairs have a name component consisting of the string "_charset_" encoded in US-ASCII, and the value component of the first such pair, when decoded as US-ASCII, is the name of a supported character encoding, then let encoding be that character encoding (replacing the default passed to the algorithm). Ouch. I hope nobody actually implements that.

tyingq 5y ago
I couldn't find support for that in urllib or python requests. But googling a bit does show it's supported in a lot of software. For example: https://bugzilla.mozilla.org/show_bug.cgi?id=18643 https://bugzilla.mozilla.org/show_bug.cgi?id=18643

the_mitsuhiko 5y ago

> In this case, he initially presumed that the problem would be with Flask, Bottle, and Tornado still honoring a semi-colon as a separator in a uri Flask (or rather Werkzeug what it's based on) removed implicit semicolon support for URL 13 years ago: https://github.com/pallets/werkzeug/commit/0ea28bbc6f5f05eefb0e648b52e6400be4b11e43 https://github.com/pallets/werkzeug/commit/0ea28bbc6f5f05eef...

tyingq 5y ago
Yeah, it appears Flask doesn't use urllib either. I think he was just outlining his approach though. That he started with popular frameworks, assuming any issues would be there, but ended up in urllib. Bottle does use parts of urllib but appears to do it's own splitting on ampersands: https://github.com/bottlepy/bottle/blob/d1413a81ead7a6f130a06e21a17ca98f4ea30df6/bottle.py#L3053 https://github.com/bottlepy/bottle/blob/d1413a81ead7a6f130a0...