5 ms·
I totally agree: The whole HTTP spec has the concept of ASCII implicitly embedded. Converting this to Unicode is, IMHO, the wrong approach. If the spec is talki
by pilif 16y ago
I totally agree: The whole HTTP spec has the concept of ASCII implicitly embedded. Converting this to Unicode is, IMHO, the wrong approach. If the spec is talking bytes, then the libraries working with that spec should also be talking bytes.
While latin1 can be converted to Unicode, this is not true for all encodings, so the current way how WSGI is handling requests might actually destroy some data in passing (some multibyte japanese encodings cannot be losslessly converted to Unicode).
If the library doesn't provide easy methods to do something, it's IMHO totally the wrong approach to potentially destroy data just to have the convenience methods available.
So the solution should be to either provide the needed library methods for bytes directly in python or to provide them as a third party library and have WSGI depend on that, though it's probably way easier to just have the internal string methods accept bytes too.
- randallsquared 16y agosome multibyte japanese encodings cannot be losslessly converted to Unicode I hadn't heard this... do you have a link or search terms that will produce more information about the issue ("convert CJK unicode" and "multibyte japanese encoding unicode" didn't)?
- pilif 16y agohttp://en.wikipedia.org/wiki/Han_unification http://en.wikipedia.org/wiki/Han_unification
- wycats 16y agoAlso, http://yehudakatz.com/2010/05/17/encodings-unabridged/ http://yehudakatz.com/2010/05/17/encodings-unabridged/ My post is mostly about the ramifications of Ruby 1.9's encoding support on Rails (a similar topic), but I go into some detail about the problems of Japanese encodings on the web.
- pilif 16y agosee also the novel I posted here on Hacker News defending the way Ruby 1.9 decided to handle strings with: http://news.ycombinator.com/item?id=1163684 http://news.ycombinator.com/item?id=1163684