2 ms·

Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (cont

by comex 10y ago

Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (contains unpaired surrogates), and explodes if you try to print it or do anything else that implicitly encodes it without the special 'surrogateescape' setting. It's really dangerous to pass such a string around. But at least you can open it, so it's not completely broken. Oh, and that means that two different strings can encode to the same path, which... is probably not usually a problem... On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post. I don't know what you mean by "corruption".

takeda 10y ago
What I meant by corruption is the program might be spewing garbage, instead of simply crashing pointing where encoding problem is. As for your other point. You're right, but there's no easy way resolving this. Either treat everything as unicode and remember it's binary on linux, or have linux working and everything else broken. In that scenario you probably should use .encode('utf-8', 'surrogateescape') as soon as you can and work on bytes, so no need of doing the above. Also, if you for example call os.listdir() with argument in bytes it'll return entries in bytes: Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.listdir(b'.') [b'\xff\xff\xff', b'test.py'] But IMO, things still could be better. I would love if there was matching sys.setfiliesystemencoding() to sys.getfilesystemencoding(). In scenarios when you are not sure of encoding you could fall back to iso-8859-1 (latin-1) which has 1:1 mapping for bytes. Currently looks like you would have to use: filename.encode(sys.getfilesystemencoding(), 'surrogateescape').decode('iso-8859-1')

by comex 10y ago

takeda 10y ago

What I meant by corruption is the program might be spewing garbage, instead of simply crashing pointing where encoding problem is. As for your other point. You're right, but there's no easy way resolving this. Either treat everything as unicode and remember it's binary on linux, or have linux working and everything else broken. In that scenario you probably should use .encode('utf-8', 'surrogateescape') as soon as you can and work on bytes, so no need of doing the above. Also, if you for example call os.listdir() with argument in bytes it'll return entries in bytes: Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.listdir(b'.') [b'\xff\xff\xff', b'test.py'] But IMO, things still could be better. I would love if there was matching sys.setfiliesystemencoding() to sys.getfilesystemencoding(). In scenarios when you are not sure of encoding you could fall back to iso-8859-1 (latin-1) which has 1:1 mapping for bytes. Currently looks like you would have to use: filename.encode(sys.getfilesystemencoding(), 'surrogateescape').decode('iso-8859-1')