12 ms·
I don't know... I get an error from the first script with python3: $ ls test test3.py test.py tøst 日本語 $ python2.7 test.py * hello hellø こに
by e12e 13y ago
I don't know... I get an error from the first script with python3:
$ ls
test test3.py test.py tøst 日本語
$ python2.7 test.py *
hello hellø こにちは tøst 日本語
import sys
# (…)
hello hellø こにちは tøst 日本語
hello hellø こにちは tøst 日本語
$ python3 test.py *
Traceback (most recent call last):
File "test.py", line 13, in <module>
shutil.copyfileobj(f, sys.stdout)
File "/usr/lib/python3.2/shutil.py", line 68, in copyfileobj
fdst.write(buf)
TypeError: must be str, not bytes
#But I can make it work with:
$ diff test.py test3.py
8c8
< f = open(filename, 'rb')
---
> f = open(filename, 'r')
$ python3 test3.py *
# same as above
Now, these two scripts are no longer the same, the python3 script
outputs text, the python2 script outputs bytes:
$ python3 test3.py /bin/ls
Traceback (most recent call last):
File "test3.py", line 13, in <module>
shutil.copyfileobj(f, sys.stdout)
File "/usr/lib/python3.2/shutil.py", line 65, in copyfileobj
buf = fsrc.read(length)
File "/usr/lib/python3.2/codecs.py", line 300, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 24: invalid start byte
The other script works like cat -- and dumps all that binary crap to the
terminal.
So, yeah, I guess things are different -- not entirely sure that the
python3 way is broken, though? It's probably correct to say that it
doesn't work well with the "old" unix way in which text was ascii and
binary was just bytes -- but consider:
$ cat /bin/ls |wc
403 2565 114032
e12e@stripe:~/tmp/python/unicodetest $ du -b /bin/ls
114032 /bin/ls
Does that "wordcount" and "linecount" from wc make any sense? For that
matter, consider:
$ cat test
hello hellø こにちは tøst 日本語
e12e@stripe:~/tmp/python/unicodetest $ wc test
1 5 42 test
(Here the word count does make sense, but just because it's an
artificial example, it wouldn't make sense for actual Japanese).
The character count is pretty certainly wrong unless you cared about
what "du -b" thinks of the number of bytes...
- lifeisstillgood 13y agoThe japanese example is interesting - because wc really rather depends on the language. So does regex. And quite a lot of other things that are useful in a Latin-derived world kind of get harder in a right to left inflected written language (if there is one, some Arabic comes to mind). I think if anything will force us to rethink the underlying assumptions of Unix, its unicode.
- e12e 13y agoPlease note that: $ echo "wc can't count æøæ either" |wc 1 5 29 $ echo "wc can't count aaa either" |wc 1 5 26 [edit: Also, note that Japanese is both left-to-right and top-to-bottom,right-to-left]
- lifeisstillgood 13y agoIs there a way to divide the Unicode world into ranges, with some clearly marked as "will work with this approach" and others marked differently. A sort of code-pages approach - but we all work on the same Unicode foundation, just when it comes to Japanese a non-speaker like me would gracefully down-scale all the operations to "print and then suggest we hire some people to write an extension". Its I guess linking LOCALE to a number range...
- EdiX 13y agowc counts bytes, to make it count characters use -m in the GNU version.
- lifeisstillgood 13y agoI think the point being made is that -m does not count characters, it counts multi-bytes. Or at least tries to. So the same Unicode point in utf-8 and utf-16 (and utf-32) could be very different strings of bytes. No way to tell unless you know before hand you are dealing with utf-8 or 16. Hence BOM, but no one likes that. Its hard. And possibly we have to abandon tools like wc when we leave the Latin world.
- the_mitsuhiko 13y agoThe first script does not work on Python 3. That's the whole point of the post. You need to use the second one.
- e12e 13y agoWell, the first script works if you don't open the files as binary files (see my "patch") -- and if you don't you get a sane error when you try to open a binary file. So they're different, but I still say it's not so clear that one is broken, one is not. I'm also not clear where all the comments on filenames comes from, but maybe they just happen to work in my case? [edit: see my other comment on treating stdin/out as binary from python3]
- DasIch 13y agoIf you don't open the files as binary, the script doesn't do what it's supposed to.
- e12e 13y agofrom the manpage of cat(1) my emphasis: "cat - concatenate files and print on the standard output" Cat has quite a few options that clearly deal with text files: --number-nonblank, --show-ends, --number, -show-tabs, --show-nonprinting, --squeeze-blank ... BSD cat[1] is a little more conservative, but it's not entirely clear cut that when printing to standard output (or reading from standard input) you want binary mode. And silently copying a binary stream to a text stream does sound like a bug. Even if that is the expected behaviour of standard "cat". [1] http://www.freebsd.org/cgi/man.cgi?cat(1) http://www.freebsd.org/cgi/man.cgi?cat(1)
- lazzlazzlazz 13y agoIsn't the entire point of the post that the first Python3 example will not work?
- e12e 13y agoYes, but I think the post is somewhat disingenuous in that if you want to read text-files, you can just open the files in non-binary (ie: the exact same script with a single character changed, the mode from "rb" to "r") -- and it will work for most cases where you can expect it to work. I've not dived all the way down the (python3) rabbit hole, but it seems to me that there probably is (should be, at least) a way to say, yes, I want to read and write in binary, I don't care about conversion, even if one or both streams happen to be standard input/output/error, not just some other binary file. I agree that if the second script indeed is the simplest way, then that is probably too hard. But as clearly demonstrated, it's not that hard to read and write "text files". [edit: to whit: "By default, these [stdin,out,err] streams are regular text streams as returned by the open() function. (...) To write or read binary data from/to the standard streams, use the underlying binary buffer. For example, to write bytes to stdout, use sys.stdout.buffer.write(b'abc'). Using io.TextIOBase.detach(), streams can be made binary by default. This function sets stdin and stdout to binary:" def make_streams_binary(): sys.stdin = sys.stdin.detach() sys.stdout = sys.stdout.detach() https://docs.python.org/3/library/sys.html#sys.stdout https://docs.python.org/3/library/sys.html#sys.stdout]
- e12e 13y agoSo, to make the script work similarly for python3: $ diff test.py test3.py 5c5 < f = sys.stdin --- > f = sys.stdin.buffer 13c13 < shutil.copyfileobj(f, sys.stdout) --- > shutil.copyfileobj(f, sys.stdout.buffer) We can now read in binary, and write to stdout in binary, and this script will happily garble your terminal if you point it at /bin/ls. If the lines above were unclear, here is the entire "python3" version. Note that we now explicitly treat standard input and output as binary (which is "wrong"): import sys import shutil for filename in sys.argv[1:]: f = sys.stdin.buffer if filename != '-': try: f = open(filename, 'rb') except IOError as err: print >> sys.stderr, 'cat.py: %s: %s' % (filename, err) continue with f: shutil.copyfileobj(f, sys.stdout.buffer) And we can now, for example: $ head -c 200 /bin/ls | python3 test3.py - ELF>�H@@p�@8 @@@@@@�88@8@@ $ The crucial difference is, that if we wanted only to deal with text, we'd get a sane error (see above) from python3. I'd probably add a comment about the sys.stdin.buffer as it isn't exactly obvious that what we do is go from stdin that is a text stream to the underlying buffer that is binary -- but I can't really agree that this is super-hard -- it took me a few minutes to google "python3 binary io" and find this… And all these scripts (appear to) deal fine with utf-8 file names both in python2 and 3. [edit: Having redone this by hand, I see that this is sort of what the "new" script in the post does, with some caveats on how stdin/out is "sometimes" not not binary... I still have a hard time accepting that this short python3 script is much more fragile than the original short python2 script. The fact that you get an error if you try to copy a binary stream to a text stream seems sane to me...]