3 ms·

there isn't an implicit bytes/unicode conversion in python2. Do you mean calling unicode('some non unicode string')? That just uses system default encoding. (

there isn't an implicit bytes/unicode conversion in python2. Do you mean calling unicode('some non unicode string')? That just uses system default encoding. ( sys.setdefaultencoding() ). Just find and replace them and slap in a .decode('UTF-8') or whatever your default encoding was in python2. Grep for the strings encode, decode, unicode and just mechanically fix them one at a time by making the old implicit behavior explicit. How many times could you be doing that anyway? A few hundred? A thousand? You could even script this pretty reliably and just page through the diff you end up with to eyeball them one at a time. I guess you might mean 'str' + u'unicodestr' or something, but again you can find these pretty easily by rooting out where the non-unicode strings are being produced and fixing the problem there. They are either literals or they are coming from IO or calls to str, right? Anyway, I've done this quite a few times and the main concern I've always had was trying to get the patch in place before people commit too much stuff for me to be able to merge the fixed up branch. Of course, you could just do it little by little by taking out places that you are relying on systemdefaultencoding by monkeypatching the default decoding function in sys to log tracebacks whenever it is used, and then whacking them as they come up so you end up with properly handled and explicit unicode decoding before you move away from python2. I bet you could find and fix 95% of the cases in a day of effort.

int_19h 7y ago
('str' + u'unicodestr') involves an implicit conversion of the bytes operand on the left side to unicode, yes, so clearly Python 2 has it. And it's not the only such case- the fundamental problem is that the Python/C API for Python 2 implements this for the standard argument parsing functions. So basically any function implemented in native code that invokes PyArg_ParseTuple("s"), will also accept Unicode objects, and will implicitly encode them with ASCII. IIRC the reverse is true for "u". So those conversions happen every time you cross from Python into native - and all built-in data types, operators, and functions are native, as are large swaths of the standard library. And yeah, u"" literals aren't hard to find, but the problem is when you get data flowing from different sources, so both sides are variables. For example, one is read from a text file, and another one comes from parsed JSON - so the former is raw bytes, and the latter is Unicode - and you need to combine them together. Like you said, the proper way to do this is to ensure that as soon as data crosses the I/O boundary, it should be of the correct type (i.e. unicode rather than bytes) - which, ironically, is exactly what Python 3 encourages with its changes. But it can be hard to find all such places - you have to actually audit every use of I/O one by one, because in Python 2, the code by itself doesn't always reflect whether it's supposed to be dealing with text or binary data.
- ltbarcly3 7y ago
  So I'm back to my idea of monkeypatching the standard libraries to add logging to a file wherever it resorts to the default encoding or an implicit decode, and one by one fix them until the log file stops having stuff in it. After a couple of months of not seeing any you just assume you found them all, and you can safely bet that you'll find the last couple in a couple more months, but by then you have managed to move to Python3 a few months ago.

int_19h 7y ago

('str' + u'unicodestr') involves an implicit conversion of the bytes operand on the left side to unicode, yes, so clearly Python 2 has it. And it's not the only such case- the fundamental problem is that the Python/C API for Python 2 implements this for the standard argument parsing functions. So basically any function implemented in native code that invokes PyArg_ParseTuple("s"), will also accept Unicode objects, and will implicitly encode them with ASCII. IIRC the reverse is true for "u". So those conversions happen every time you cross from Python into native - and all built-in data types, operators, and functions are native, as are large swaths of the standard library. And yeah, u"" literals aren't hard to find, but the problem is when you get data flowing from different sources, so both sides are variables. For example, one is read from a text file, and another one comes from parsed JSON - so the former is raw bytes, and the latter is Unicode - and you need to combine them together. Like you said, the proper way to do this is to ensure that as soon as data crosses the I/O boundary, it should be of the correct type (i.e. unicode rather than bytes) - which, ironically, is exactly what Python 3 encourages with its changes. But it can be hard to find all such places - you have to actually audit every use of I/O one by one, because in Python 2, the code by itself doesn't always reflect whether it's supposed to be dealing with text or binary data.

ltbarcly3 7y ago
So I'm back to my idea of monkeypatching the standard libraries to add logging to a file wherever it resorts to the default encoding or an implicit decode, and one by one fix them until the log file stops having stuff in it. After a couple of months of not seeing any you just assume you found them all, and you can safely bet that you'll find the last couple in a couple more months, but by then you have managed to move to Python3 a few months ago.