4 ms·
In the Reddit discussion of this, someone linked to this criticism [1] of Python 3's Unicode handling written by Armin Ronacher, author of the Flask framework.
by tzs 11y ago
In the Reddit discussion of this, someone linked to this criticism [1] of Python 3's Unicode handling written by Armin Ronacher, author of the Flask framework.
I am not competent to say whether this is spot on or rubbish or somewhere in between [2], but it seemed interesting at least.
[1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/ http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/
[2] Almost all of my Python 2 experience is in homework assignments in MOOCs for problems where there was no need to care about whether strings were ASCII, UTF-8, binary, or something else. My Python 3 experience is a handful of small scripts in environments where everything was ASCII.
- lmm 11y agoIt's wrong. The whole point of cat is to concatenate files. But if you concatenate two files with different encodings, you end up with an unreadable file. So you want cat to error out if one of the files that was passed has a different encoding from the encoding you told it to use, which is exactly what the python cat will do.
- jlg23 11y ago> The whole point of cat is to concatenate files. Yes. > So you want cat to error out if one of the files that was passed has a different encoding. No. I expect it to read bits from stream A until it is exhausted and then read bits from stream B until it is exhausted. All the time just writing the ones and zeros read to the output stream (of bits). And no, a byte does not have to be "8 bit" (http://www.lispworks.com/documentation/HyperSpec/Body/f_by_by.htm http://www.lispworks.com/documentation/HyperSpec/Body/f_by_b...).
- cosarara97 11y agoDo you actually work with systems not using 8 bit bytes or are you just being pedantic?
- jlg23 11y agoYes, I do. But the point I was trying to make was "it's bits, not bytes" (because "byte" has been defined to mean 8 bits only recently).
- lmm 11y agoAnd concatenating a file of 9-bit bytes with a file of 8-bit bytes will produce something useful? No. If you don't know what your bits represent then you will corrupt them. Python does not need to faithfully reproduce all the historical oddities of unix.
- jlg23 11y agoIt might - depending on what I intend to do with the file (I still have the offsets of the individual files because I know the original file sizes). API-wise, in my humble experience, it's hell to deal with operations that are supposed to work on bit-streams but try to be smart and ask me for encodings of those - this is information I might not even have when building on those basic operations. The "oddities", how you call them, are the result of not over-abstracting the code to handle yet-unknown problems. You want to concatenate text files with different encodings? Convert them. Expecting a basic tool to do this for you (or carp on "problems") quintessentially leads to cat converting image formats for you and demanding to know how to combine those images: (addition, subtraction, append to left/top/right/bottom of first image etc).
- lmm 11y ago> API-wise, in my humble experience, it's hell to deal with operations that are supposed to work on bit-streams but try to be smart and ask me for encodings of those - this is information I might not even have when building on those basic operations. The "oddities", how you call them, are the result of not over-abstracting the code to handle yet-unknown problems. That's how C ended up as the security nightmare that it is. There are a lot of things you can do in C that you can't do in Python - you can reinterpret strings or floats as integers, you can subtract pointers from integers, you can write to random memory addresses.... Sometimes these things are useful, but most of the time they just lead to your program breaking in an unhelpful, nonobvious way. Python is not that kind of language; it will go to some pains to help you not make mistakes. If you want to do bit-twiddling in Python there are APIs for it, and you could implement a "bit-level cat" using them, but it's never going to be the primary use case. Arguably there should be better support for accessing stdin/stdout in binary mode, but that would make it very easy to accidentally interleave text and binary output which would again result in (probably silent) corruption. (Writing a "binary cat" that concatenates two files of bytes would not lead to any of the problems in the linked article - it's only trying to use stdin/stdout that's causing the trouble in the link).
- mark-r 11y agoIt seems simple to me - if you want bytes, open the file in binary mode, if you want strings open it in text mode. The only glitch is with stdin/stdout. They're opened on your behalf before your program even starts, and the assumption is that you'll be reading and writing text in the default OS encoding. This doesn't mesh well with the Unix pipe paradigm.
- pdonis 11y agoIf you need binary reads/writes for stdin/stdout, you can use the underlying binary buffer objects (the "buffer" attributes on sys.stdin/stdout).