5 ms·
> That I work with "boundary code" so obviously that's harder on Python 3 now (duh) mhm. I tell people now and then that python 3 (and the python 3 developers)
by shadowmint 12y ago
> That I work with "boundary code" so obviously that's harder on Python 3 now (duh)
mhm. I tell people now and then that python 3 (and the python 3 developers) are hostile to people embedding it and using it for low level tasks specifically because of this unicode stuff, and they tend to tell me I should just suck it up.
I suppose I'm morbidly glad not the only one feeling the pain, but really, it honestly feels like python 3 line is just not making any effort towards making this stuff easier and simpler. :/
- ygra 12y agoUnicode, dealing with text, i18n are never easy and simple. That being said, there are lots of things that work on both Windows and Unix and use Unicode internally, even for file names and paths (e.g. Qt and the already-mentioned Java). Qt is even used by a popular-ish desktop environment. If that approach were that unsuitable and utterly incompatible with the Unix approach on encodings I wonder why it apparently does work.
- tormeh 12y agoIsn't utf-8 just a string of bytes where each byte represents a sign? Why is it hard?
- sp332 12y agoNo. http://en.wikipedia.org/wiki/UTF-8#Description http://en.wikipedia.org/wiki/UTF-8#Description How could you encode millions of characters into just 256 values?
- tormeh 12y agoHonestly, I thought that was what utf-16 was for. I thought the number was the bitlength of a sign.
- ygra 12y agoThe number says how wide a code unit is. However, it doesn't say anything about how many code units are required to encode a single code point. UTF-8 needs 1–4, UTF-16 needs 1–2 and UTF-32 needs 1. They all are able to represent all encoded characters of Unicode, it's just that the individual bytes are different.
- _ZeD_ 12y agobecause is not.
- kalleboo 12y agoNo, single characters can be represented by a string of multiple bytes. And you can construct an invalid string of bytes that would cause a Unicode error. The problem comes from what to you in these applications when you come across a Unicode error. Such as, if the user tries to open a file where the filename contains a string of bytes that would be invalid Unicode. Refuse to open the file?
- tormeh 12y agoIsn't the standard approach to just show a box or similar gibberish? I've seen enough of those.
- ygra 12y agoBoxes are just the font not having that glyph – that doesn't destroy data. Question marks, U+FFFD or random gibberish are arguably at least as disastrous as refusing to open a file (maybe more because you don't actually notice something is wrong).
- sp332 12y agoCould you use the "bytes" type instead of the "string" type for that low-level stuff?
- andreasvc 12y agoThat would be preferable in these situations. However, the problem is that the standard library uses Unicode for filenames and stdin/out by default, as described in the blog post, and you get errors as soon as anything cannot be decoded.
- sp332 12y agoAw, I knew it couldn't be that easy.