5 ms·
> I found code all over the place which broke for unicode reasons (we were reading in the output of GCC, and printing it back out, and it didn't work any more
by ric2b 8y ago
> I found code all over the place which broke for unicode reasons (we were reading in the output of GCC, and printing it back out, and it didn't work any more as GCC doesn't output valid UTF8 in some cases).
That means the code was working with raw bytes, not utf-8 strings, so that's what you should convert to on Python3.
That means using `.encode()` and `.decode()` or using bytestrings.
Python3 doesn't break string usage, it makes you do it correctly. Expect a lot more of that kind of pain by moving to Rust (not saying it's not worth, you'll indeed get more correct programs done).
- CJefferson 8y agoThe problem I had with python is that (because it is dynamically typed), and quite a few methods take both bytes and strings, it can be tricky to track what you have and often things don't break until you feed in a non-UTF8 string. I found it much easier in Rust, as the two types are just distinct, and an incorrect program just fails to compile.
- orf 8y agoIn Python 3 they are two distinct types as well.
- lmm 8y agoThey're not really "types" when you only find out at runtime.
- orf 8y agoSure they are.
- joshuamorton 8y agoWhen handling tricky string issues, my first step is usually to explicitly type annotate the code (using mypy annotations). This makes string handling and tracking much simpler.