3 ms·

In this example the dichotomy is between String (which is guaranteed by the type system to be valid UTF-8) and OsStr (which might be in an unknown encoding or o

In this example the dichotomy is between String (which is guaranteed by the type system to be valid UTF-8) and OsStr (which might be in an unknown encoding or otherwise not decodable to valid Unicode). This is exactly when you want a systems language to require explicit conversions, rather than converting things silently and possibly losing or corrupting data.

davvid 12y ago
rather than converting things silently and possibly losing or corrupting data Exactly. Python3 went down the "silently converting" route, and it's not pretty[1]. I would go so far as to call it harmful. http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ I understand the difficulty in this space; much of it is caused by forcing the Windows unicode filesystem API onto python as its world-view, rather than sticking to the traditional Unix bytes world-view. I'm unixy, so I'm completely biased, but I think adopting the Windows approach is fundamentally broken.
- Veedrac 12y ago
  The problem there is overblown - it's basically all due to the idea that sys.stdin or sys.stdout might get replaced with streams that don't have a binary buffer. The simple solution is just not to do that (and it's pretty easy; instead of replacing with a StringIO, replace it with a wrapped binary buffer). Then the code is quite simple import sys import shutil for filename in sys.argv[1:]: if filename != '-': try: f = open(filename, 'rb') except IOError as err: msg = 'cat.py: {}: {}\n'.format(filename, err) sys.stderr.buffer.write(msg.encode('utf8', 'surrogateescape')) continue else: f = sys.stdin.buffer with f: shutil.copyfileobj(f, sys.stdout.buffer) Python's surrogateescape'd strings aren't the best solution, but I personally believe that treating unicode output streams as binary ones is even worse.

davvid 12y ago

rather than converting things silently and possibly losing or corrupting data Exactly. Python3 went down the "silently converting" route, and it's not pretty[1]. I would go so far as to call it harmful. http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ I understand the difficulty in this space; much of it is caused by forcing the Windows unicode filesystem API onto python as its world-view, rather than sticking to the traditional Unix bytes world-view. I'm unixy, so I'm completely biased, but I think adopting the Windows approach is fundamentally broken.

Veedrac 12y ago
The problem there is overblown - it's basically all due to the idea that sys.stdin or sys.stdout might get replaced with streams that don't have a binary buffer. The simple solution is just not to do that (and it's pretty easy; instead of replacing with a StringIO, replace it with a wrapped binary buffer). Then the code is quite simple import sys import shutil for filename in sys.argv[1:]: if filename != '-': try: f = open(filename, 'rb') except IOError as err: msg = 'cat.py: {}: {}\n'.format(filename, err) sys.stderr.buffer.write(msg.encode('utf8', 'surrogateescape')) continue else: f = sys.stdin.buffer with f: shutil.copyfileobj(f, sys.stdout.buffer) Python's surrogateescape'd strings aren't the best solution, but I personally believe that treating unicode output streams as binary ones is even worse.