7 ms·
So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points. The main breaking change is that
by kstrauser 11y ago
So, it's like Python. You can still mix and match; you just have to explicitly convert between them at the appropriate points.
The main breaking change is that you can't read a stream of bytes from a byte-oriented resource (say, a network connection that by definition can only send a stream of octets) and magically treat it like text elsewhere. While that's a tempting thing to want to do, it's been an unending fountain of bugs over the years. For example, in Python 2.7:
>>> a = 'this is a test ಠ_ಠ'
>>> b = bytes(a)
>>> b
'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0'
>>> type(b)
<type 'str'>
In Python 3.5:
>>> a = 'this is a test ಠ_ಠ'
>>> print(a)
this is a test ಠ_ಠ
>>> b = bytes(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
>>> b = bytes(a.encode('utf-8'))
>>> b
b'this is a test \xe0\xb2\xa0_\xe0\xb2\xa0'
>>> type(b)
<class 'bytes'>
In 2.7, you encode a series of Unicode values back to... a string? This is the source of all kinds of fun as functions expecting a string as input don't have a way to tell whether it's already been encoded or if they need to do so. In 3.5, there's no ambiguity. Functions that write to files or network sockets can either receive strings and know that they have to encode the strings locally, or bytes and know that they don't have to.
Right up near the top of Zen we get:
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
The Python 2-style implicit conversions were awfully convenient 15 years ago when most things were ASCII. They're absolutely painful now that we've moved past the expectation that strings are arrays of single bytes.
- BuckRogers 11y agoExcept Python2 is the POSIX standard on how to handle this and I prefer retro OSs without a bytes/string distinction. Python3 is the Windows way of handling strings so the change was pure churn. Both are valid, but Linux is "pretty popular" right? So why break everyone's code just to churn the pot. If it was good for Kernigan and Richie, good for Joe Ossanna, good for Robert Pike well it's good enough for me.
- klodolph 11y agoThere isn't a "Windows" way of handling strings. You can handle strings however you want, just make sure that you pass the right data to syscalls. Windows APIs take UTF-16 or some other code page. Linux takes bytes. OS X takes UTF-8. There is no greatest common denominator. But if you use Unicode, then at least there's a well-defined way to write the syscall wrappers on all platforms. And it's not like Python 3 is forcing you to give up bytes. You can still os.listdir(b'/some/dir') and you'll get a list of bytes objects back. It's just that that's not default, since using Unicode by default simply makes more sense on a cross-platform system.
- jowiar 11y agoThe problem is it might be good enough for Kernighan and Ritchie, but it's not good enough for Chávez and Çelik (or, well, most of the world). The A in ASCII stands for "American" -- and it gives a good indication of where you can look to find a whole lot of folks who ASCII isn't good enough for.
- true_religion 11y agoActually I'd have naively thought it meant America as in "The Americans" including South America. It's no wonder the IANA want's to change it into US-ASCII (or USA-SCII).
- BuckRogers 11y agoPython2 supports unicode. Chávez and Çelik are covered. I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform. Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).
- vetinari 11y ago> Python2 supports unicode. Chávez and Çelik are covered. Most python2 code doesn't. It has a nice property of exploding spectacularly when the string you are writing out to console or file contains Chávez or Çelik.
- the_mitsuhiko 11y ago> Most python2 code doesn't. Citation needed. I can't say I ever had practical issues with not being able to handle unicode on Python if I needed. No matter which library. Please give practical examples of where you cannot deal with unicode on Python 2.
- vetinari 11y agojust plain for line in yourfile: print line with yourfile containing non-ascii characters and running it in windows console.
- toyg 11y ago> So why break everyone's code Dunno about you, but the amount of encoding/decoding errors in Py2 broke my little code multiple times in the past. "oh yeah I'll just fetch this html page"... all good, until that page contains non-ASCII and then BOOM. With Py3 you're forced to think about this sort of issues right away and with minimal fuss you're sorted for life.
- deleted 11y ago[deleted]
- tomjen3 11y agoIt may just be me, but I tend to assume all strings are UTF-8.
- kstrauser 11y agoThen I think you'd be much happier with Python 3 as it's explicitly clear when you're transforming between str and bytes objects.
- mark-r 11y agoI think you miss the point - if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. By rubbing your nose in the difference between str and bytes, Python 3 complicates your life. On the flip side it makes the complicated case more tractable.
- mkesper 11y agoThere is nothing more mind-boggling than using mixed sources of ascii and unicode in Python2.
- kstrauser 11y agoBut they are different animals. When you read from a socket, you get bytes. Are they text or are they a JPG? Py2 says "they're text holding the contents of a JPG." Py3 says "I don't know. It's your data, you tell me."
- comex 11y agoWhich works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. For most strings there's no actual need to analyze them as text or look at character properties or do anything other than move them from one place to another, possibly interpolated with other strings, and in that case byte strings work fine, including with non-English text if everything standardizes on UTF-8. Therefore they should be the default, and explicit Unicode processing the exception, like Py2. The only situation where explicit decoding improves things is if you have to deal with legacy 8-bit encodings, which aren't all that common these days, and in that case you probably should be doing complicated things anyway like parsing any applicable headers, autodetecting encoding if that doesn't work, giving the user an override option, etc., which is work that types can't save you from. Using byte strings also doesn't 'just work' if the original format is a Unicode encoding other than UTF-8; e.g. Windows filenames are UTF-16 and can't be converted to UTF-8 without risking errors caused by unpaired surrogates. But that's what WTF-8 is for (or rather, what it should be for, disclaimers in the spec notwithstanding)...