5 ms·

I think you miss the point - if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. By rubb

by mark-r 10y ago

I think you miss the point - if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. By rubbing your nose in the difference between str and bytes, Python 3 complicates your life. On the flip side it makes the complicated case more tractable.

mkesper 10y ago
There is nothing more mind-boggling than using mixed sources of ascii and unicode in Python2.
kstrauser 10y ago
But they are different animals. When you read from a socket, you get bytes. Are they text or are they a JPG? Py2 says "they're text holding the contents of a JPG." Py3 says "I don't know. It's your data, you tell me."
- comex 10y ago
  Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. For most strings there's no actual need to analyze them as text or look at character properties or do anything other than move them from one place to another, possibly interpolated with other strings, and in that case byte strings work fine, including with non-English text if everything standardizes on UTF-8. Therefore they should be the default, and explicit Unicode processing the exception, like Py2. The only situation where explicit decoding improves things is if you have to deal with legacy 8-bit encodings, which aren't all that common these days, and in that case you probably should be doing complicated things anyway like parsing any applicable headers, autodetecting encoding if that doesn't work, giving the user an override option, etc., which is work that types can't save you from. Using byte strings also doesn't 'just work' if the original format is a Unicode encoding other than UTF-8; e.g. Windows filenames are UTF-16 and can't be converted to UTF-8 without risking errors caused by unpaired surrogates. But that's what WTF-8 is for (or rather, what it should be for, disclaimers in the spec notwithstanding)...
  - takeda 10y ago
    > Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. I'm tired of people spreading FUD about it. they either don't understand what they are doing (and do it incorrectly) or are repeating what other people (who did things incorrectly) said: » echo $LANG en_US.UTF-8 » echo $LC_CTYPE en_US.UTF-8 » echo 'works' > $'\377'$'\377'$'\377' » ls ??? » cat $'\377'$'\377'$'\377' works » python3 Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = b'\377\377\377' >>> a b'\xff\xff\xff' >>> a.decode() Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte >>> b = open(a) >>> b.read() 'works\n' >>> Accessing files with invalid utf-8 characters works just fine. Being explicit and clearly distinguishing between a string and bytes allows python to know when the filename is encoded using utf-8 (or in fact any other encoding you have defined in your environment) or bytes. Actually python2 struggles here, because it treats everything as bytes and can't tell what is used. Yes, legacy python probably won't crash, but it will spew garbage. Python3 on the other side is strict about to point errors in your code. If python3 code will crash it's either bug in python or (most likely) bug in your code and you should fix it. I (and I'm sure a lot of people) prefer program to crash, pointing where the error is, than silently corrupt data once in a while, and when you notice the issue spending days or months figuring out where the corruption is happening.
    - comex 10y ago
      Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (contains unpaired surrogates), and explodes if you try to print it or do anything else that implicitly encodes it without the special 'surrogateescape' setting. It's really dangerous to pass such a string around. But at least you can open it, so it's not completely broken. Oh, and that means that two different strings can encode to the same path, which... is probably not usually a problem... On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post. I don't know what you mean by "corruption".
devishard 10y ago
> if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. Let me guess: you don't work on anything that has any internationalization ever.
- mark-r 10y ago
  Make a list of the things that work vs. the things that don't. Works: input, output, copying, formatting, numeric conversion. Doesn't: slicing, character counting. Probably doesn't: regular expressions, case conversion. Remember that all this is premised on all I/O being done in UTF-8.
  - devishard 10y ago
    So? Even if we accept your myopic list of features that work and believe your statement that "most operations can be performed on bytes", that doesn't make the language usable for internationalized content. Trading having to call encode and decode, which fail with clear, easy-to-fix errors, for a myriad of subtle, silent encoding failures is not a good trade. Which you would be abundantly clear to you if you had internationalized anything ever.
  - ubernostrum 10y ago
    During my last job hunt I kept getting people doing the "test for palindrome" phone-screen question. After a while I started creatively expressing my dislike of phone screens by showing them an implementation that hints at some of the complexity lurking in this allegedly-simple task (i.e., you need to understand Unicode classes and normalization in order to write a palindrome tester).
    - devishard 10y ago
      That's totally a question I'd ask, and I don't understand why you don't like it. The reason I'd ask it is not because it's a simple task, it's because it's not a simple task: what I'm looking for is that you aren't going to over-confident-ly throw naive solutions at hard problems. That will cause bugs and have to be cleaned up. I think if you showed me an implementation that hinted at some of the complexity lurking in the task, that would be a big point in your favor.
ubernostrum 10y ago
if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character. By rubbing your nose in the difference between str and bytes, Python 3 complicates your life. Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.
- bubuga 10y ago
  > In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character. Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?
  - ubernostrum 10y ago
    If the main feature is it's variable-length, why would any programmer assume the length doesn't vary? Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.
- tomjen3 10y ago
  I am a Dane - we have three letters in our alphabet that are not represented in ascii (æ, ø and å) so let me assure you I don't except text to be in ascii, but I do expect it to be in UTF-8 and will consider it a bug if it isn't.
  - Too 10y ago

by mark-r 10y ago

mkesper 10y ago

There is nothing more mind-boggling than using mixed sources of ascii and unicode in Python2.

kstrauser 10y ago

But they are different animals. When you read from a socket, you get bytes. Are they text or are they a JPG? Py2 says "they're text holding the contents of a JPG." Py3 says "I don't know. It's your data, you tell me."

comex 10y ago
Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. For most strings there's no actual need to analyze them as text or look at character properties or do anything other than move them from one place to another, possibly interpolated with other strings, and in that case byte strings work fine, including with non-English text if everything standardizes on UTF-8. Therefore they should be the default, and explicit Unicode processing the exception, like Py2. The only situation where explicit decoding improves things is if you have to deal with legacy 8-bit encodings, which aren't all that common these days, and in that case you probably should be doing complicated things anyway like parsing any applicable headers, autodetecting encoding if that doesn't work, giving the user an override option, etc., which is work that types can't save you from. Using byte strings also doesn't 'just work' if the original format is a Unicode encoding other than UTF-8; e.g. Windows filenames are UTF-16 and can't be converted to UTF-8 without risking errors caused by unpaired surrogates. But that's what WTF-8 is for (or rather, what it should be for, disclaimers in the spec notwithstanding)...
- takeda 10y ago
  > Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'. I'm tired of people spreading FUD about it. they either don't understand what they are doing (and do it incorrectly) or are repeating what other people (who did things incorrectly) said: » echo $LANG en_US.UTF-8 » echo $LC_CTYPE en_US.UTF-8 » echo 'works' > $'\377'$'\377'$'\377' » ls ??? » cat $'\377'$'\377'$'\377' works » python3 Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux Type "help", "copyright", "credits" or "license" for more information. >>> a = b'\377\377\377' >>> a b'\xff\xff\xff' >>> a.decode() Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte >>> b = open(a) >>> b.read() 'works\n' >>> Accessing files with invalid utf-8 characters works just fine. Being explicit and clearly distinguishing between a string and bytes allows python to know when the filename is encoded using utf-8 (or in fact any other encoding you have defined in your environment) or bytes. Actually python2 struggles here, because it treats everything as bytes and can't tell what is used. Yes, legacy python probably won't crash, but it will spew garbage. Python3 on the other side is strict about to point errors in your code. If python3 code will crash it's either bug in python or (most likely) bug in your code and you should fix it. I (and I'm sure a lot of people) prefer program to crash, pointing where the error is, than silently corrupt data once in a while, and when you notice the issue spending days or months figuring out where the corruption is happening.
  - comex 10y ago
    Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (contains unpaired surrogates), and explodes if you try to print it or do anything else that implicitly encodes it without the special 'surrogateescape' setting. It's really dangerous to pass such a string around. But at least you can open it, so it's not completely broken. Oh, and that means that two different strings can encode to the same path, which... is probably not usually a problem... On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post. I don't know what you mean by "corruption".

devishard 10y ago

> if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever. Let me guess: you don't work on anything that has any internationalization ever.

mark-r 10y ago
Make a list of the things that work vs. the things that don't. Works: input, output, copying, formatting, numeric conversion. Doesn't: slicing, character counting. Probably doesn't: regular expressions, case conversion. Remember that all this is premised on all I/O being done in UTF-8.
- devishard 10y ago
  So? Even if we accept your myopic list of features that work and believe your statement that "most operations can be performed on bytes", that doesn't make the language usable for internationalized content. Trading having to call encode and decode, which fail with clear, easy-to-fix errors, for a myriad of subtle, silent encoding failures is not a good trade. Which you would be abundantly clear to you if you had internationalized anything ever.
- ubernostrum 10y ago
  During my last job hunt I kept getting people doing the "test for palindrome" phone-screen question. After a while I started creatively expressing my dislike of phone screens by showing them an implementation that hints at some of the complexity lurking in this allegedly-simple task (i.e., you need to understand Unicode classes and normalization in order to write a palindrome tester).
  - devishard 10y ago
    That's totally a question I'd ask, and I don't understand why you don't like it. The reason I'd ask it is not because it's a simple task, it's because it's not a simple task: what I'm looking for is that you aren't going to over-confident-ly throw naive solutions at hard problems. That will cause bugs and have to be cleaned up. I think if you showed me an implementation that hinted at some of the complexity lurking in the task, that would be a big point in your favor.

ubernostrum 10y ago

if all text is UTF-8, then most operations can be performed on bytes without worrying about any transformations whatsoever This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character. By rubbing your nose in the difference between str and bytes, Python 3 complicates your life. Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.

bubuga 10y ago
> In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character. Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?
- ubernostrum 10y ago
  If the main feature is it's variable-length, why would any programmer assume the length doesn't vary? Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.
tomjen3 10y ago
I am a Dane - we have three letters in our alphabet that are not represented in ascii (æ, ø and å) so let me assure you I don't except text to be in ascii, but I do expect it to be in UTF-8 and will consider it a bug if it isn't.
- Too 10y ago