3 ms·

Unfortunately unicode in Python 3 is not necessarily better. See http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/e

by whyever 11y ago

Unfortunately unicode in Python 3 is not necessarily better. See http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/.

raiph 11y ago
Both Perl 5 and Python 3 are built with the assumption that a "character" is what the Unicode standard generally refers to as a "character". This is a serious problem and will I suspect one day be understood to be the biggest long-term mistake made when designing Python 3. I wrote a comment about this a few months back at https://news.ycombinator.com/item?id=9791211 https://news.ycombinator.com/item?id=9791211 (see second half of the initial comment and then the followups).
jandrese 11y ago
If you are anal about getting every edge case correct, Unicode is a straight up nightmare to support. Even in Perl's supposedly superior support has issues. See how deep the rabbit hole goes by reading the first response to the question here: https://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default https://stackoverflow.com/questions/6162484/why-does-modern-... There is really no way for a simple program to account for all of those possible edge cases. Something as simple as a print of a string can be a minefield depending on what is in those characters. And it's not like the old days where you could safely filter out all non-printable characters to avoid most problems. Have you considered how your formatting is going to look when people intersperse Right to Left words in your output for example?
- bmn_ 11y ago
  > There is really no way for a simple program to account for all of those possible edge cases. The post to linked you contained a piece of boiler-plate code which accounts for said cases. > Have you considered how your formatting is going to look when people intersperse Right to Left words in your output for example? There are modules to handle this. It of course means that you have to stop using non-Unicode-aware functions like `sprintf`.
  - jandrese 11y ago
    The boilerplate on there is only a start. Any time you read or write a Unicode string, or do any manipulation of the string, you need to consider possible edge cases, and they can be nightmarish. Consider something as simple as outputting zero padded numbers (so they form a nice column in the output), except that the numbers might not be Arabic, zero might not be 0, they might not be written in the direction you expect, and padding might not even make sense. This is how you go crazy.

by whyever 11y ago

Unfortunately unicode in Python 3 is not necessarily better. See http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/.

raiph 11y ago

Both Perl 5 and Python 3 are built with the assumption that a "character" is what the Unicode standard generally refers to as a "character". This is a serious problem and will I suspect one day be understood to be the biggest long-term mistake made when designing Python 3. I wrote a comment about this a few months back at https://news.ycombinator.com/item?id=9791211 https://news.ycombinator.com/item?id=9791211 (see second half of the initial comment and then the followups).

jandrese 11y ago

If you are anal about getting every edge case correct, Unicode is a straight up nightmare to support. Even in Perl's supposedly superior support has issues. See how deep the rabbit hole goes by reading the first response to the question here: https://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default https://stackoverflow.com/questions/6162484/why-does-modern-... There is really no way for a simple program to account for all of those possible edge cases. Something as simple as a print of a string can be a minefield depending on what is in those characters. And it's not like the old days where you could safely filter out all non-printable characters to avoid most problems. Have you considered how your formatting is going to look when people intersperse Right to Left words in your output for example?

bmn_ 11y ago
> There is really no way for a simple program to account for all of those possible edge cases. The post to linked you contained a piece of boiler-plate code which accounts for said cases. > Have you considered how your formatting is going to look when people intersperse Right to Left words in your output for example? There are modules to handle this. It of course means that you have to stop using non-Unicode-aware functions like `sprintf`.
- jandrese 11y ago
  The boilerplate on there is only a start. Any time you read or write a Unicode string, or do any manipulation of the string, you need to consider possible edge cases, and they can be nightmarish. Consider something as simple as outputting zero padded numbers (so they form a nice column in the output), except that the numbers might not be Arabic, zero might not be 0, they might not be written in the direction you expect, and padding might not even make sense. This is how you go crazy.