3 ms·
('str' + u'unicodestr') involves an implicit conversion of the bytes operand on the left side to unicode, yes, so clearly Python 2 has it. And it's not the only
by int_19h 7y ago
('str' + u'unicodestr') involves an implicit conversion of the bytes operand on the left side to unicode, yes, so clearly Python 2 has it. And it's not the only such case- the fundamental problem is that the Python/C API for Python 2 implements this for the standard argument parsing functions. So basically any function implemented in native code that invokes PyArg_ParseTuple("s"), will also accept Unicode objects, and will implicitly encode them with ASCII. IIRC the reverse is true for "u". So those conversions happen every time you cross from Python into native - and all built-in data types, operators, and functions are native, as are large swaths of the standard library.
And yeah, u"" literals aren't hard to find, but the problem is when you get data flowing from different sources, so both sides are variables. For example, one is read from a text file, and another one comes from parsed JSON - so the former is raw bytes, and the latter is Unicode - and you need to combine them together. Like you said, the proper way to do this is to ensure that as soon as data crosses the I/O boundary, it should be of the correct type (i.e. unicode rather than bytes) - which, ironically, is exactly what Python 3 encourages with its changes. But it can be hard to find all such places - you have to actually audit every use of I/O one by one, because in Python 2, the code by itself doesn't always reflect whether it's supposed to be dealing with text or binary data.
- ltbarcly3 7y agoSo I'm back to my idea of monkeypatching the standard libraries to add logging to a file wherever it resorts to the default encoding or an implicit decode, and one by one fix them until the log file stops having stuff in it. After a couple of months of not seeing any you just assume you found them all, and you can safely bet that you'll find the last couple in a couple more months, but by then you have managed to move to Python3 a few months ago.