5 ms·

Yes, please make UTF-8 default! It's 2018 print(str(b'I DONT WANT TO add "UTF-8" everywhere!!','UTF-8'))

by singularity2001 9y ago

ubernostrum 9y ago
As a serialization and transmission format, UTF-8 is fine. As the basis of a high-level language's string type, UTF-8 is objectively incorrect. Strings in a high-level language should be as clean an abstraction of Unicode as possible, and leaking implementation details of the particular byte-encoding scheme up to the programmer is not acceptable.
- singularity2001 9y ago
  Ok, is it reasonable to ask for all string getter/setter file read/write ops to default to UTF-8? also having different string methods .byte_length .char_length(?) .codepoint_length seems a good idea. or string.length(aspect=bytes) but what should be the default aspect?
  - mjevans 9y ago
    For compatibility reasons, len(unicode type) should probably return the size if accessed as a raw 8 bit sequence, so an alias to .length_bytes. I also think the other names should similarly start with .length_ .
    - mark-r 9y ago
      len() should return the length in individual addressable units, so that obj[len(obj)-1] always returns the last item, whether that's a byte or a unicode character.
  - ubernostrum 9y ago
    I'm going to quote something I wrote a couple years ago[1]: Now, I should point out here that I’m not really knocking the people who were writing, say, command-line and file-handling utilities in Python. For years, Python sort of accepted the status quo of the Unix world, which was mostly to stick its fingers in its ears and shout LA LA LA I CAN’T HEAR YOU I’M JUST GOING TO SET LC_CTYPE TO C AGAIN AND GO BACK TO MY HAPPY PLACE. A bit later on it changed to “just use UTF-8 everywhere, UTF-8 is perfectly safe”, which really meant “just use UTF-8 everywhere because we can continue pretending it’s ASCII up until the time someone gives us a non-ASCII or multi-byte character, at which point do the fingers-in-ears-can’t-hear-you thing again”. So a lot of what you’ll see in terms of complaints about string handling are really complaints that Unix’s pretend-everything-is-ASCII-until-it-breaks approach was never very good to begin with and just gets worse with every passing year. I stand by this: we had a couple of decades of Python catering to this brokenness, and it made life miserable for everyone who didn't work in that particular domain. Python 3 changed that. Does it mean life got harder for some people? Yup. But life got a lot easier and more reliable for many more people, and it's a tradeoff I'm willing to accept. [1] https://www.b-list.org/weblog/2016/jun/10/python-3-again/ https://www.b-list.org/weblog/2016/jun/10/python-3-again/
    - mjevans 9y ago
      Can you answer what ways things got easier that are /not/ equal or better in the solution I proposed?
      - viraptor 9y ago
        Currently in python you know that you're holding either raw bytes, or something that can be successfully serialised to utf8. With your proposed solution you'll find out which one it is when you try to encode it. It's the difference between easier to debug: "you asked me to read a value here, but your assumptions about the encoding don't match reality" exception and the hard to debug: "I did a lot of processing; you thought this thing is a valid text, but it isn't; have fun tracking down how it got here in the first place" exception.
        mjevans 9y ago
        Python 3: We have two things that are ALMOST the same, and which if we'd done it correctly, could have been converted just by changing what we're willing to call it (in the "downgrade" direction; or also verifying if you want to achieve what I hate that Python 3 is forcing on programmers). Proposed "string like" Object: __IF__ you want to turn on debugging, sure, force it to validate assumptions at runtime/compile time. Otherwise call verify() when you're willing to handle that being some result indicating "we have a problem". Maybe the verify() call returns the byte-access-offset of the first non-conforming sequence.
        viraptor 9y ago
        > We have two things that are ALMOST the same, I think we've got a fundamental disagreement here. I don't believe they're similar at all. They just happened to get confused a lot in the past when it didn't matter that much.
      - ubernostrum 9y ago
        Which "solution" was that? Your comment suggesting that len() should be the length in bytes of a UTF-8 representation of the string? That would be a terrible thing for people who work with strings as strings, and would essentially tell such people -- who make up a lot of Python's user base! -- "go use something else, this language is only for sysadmins to write Unix utilities now".
  - mark-r 9y ago
    Unfortunately defaulting to UTF-8 is often a bad choice in Windows. Curse Microsoft!
- mjevans 9y ago
  What's unacceptable is forcing the programmer to deal with the possibility of 'garbage' strings by default. Also having major issues when half of the libraries want 'binary string like sequences' and the other half want 'native Unicode strings'. That's what the support for a generic array type, which MIGHT be handled like a string (if the programmer chooses), and which when designated as being in a Unicode format of ANY type, and processed with data of any valid Unicode type, will automatically coerce together the correct way, and produce a valid result* (assuming valid inputs, and which might still need to actually be validated (again, bitflag cleared), if anyone cares). THAT is what a high level language should be. Python has actually __REGRESSED__ in that respect. It's handing of Unicode is //low// level. It forces the programmer to need to care about the data types, and deal with legacy invalid data strings in some method that's up to them. Except they can't /ignore/ invalid sequences, because that option isn't supported.
viraptor 9y ago
Not sure what you mean. It's the default for a long time now. https://docs.python.org/3/library/stdtypes.html#bytes.decode https://docs.python.org/3/library/stdtypes.html#bytes.decode `open` also defaults to your system encoding, which is likely utf8 judging from the message.
- singularity2001 9y ago
  print(str(b'I am a broken string')) print(str(b'I DONT WANT TO add "UTF-8" everywhere!!','UTF-8')) not quite default!
  - orf 9y ago
    Explicit is better than implicit. The first line isnt a string, it's a collection of bytes, guessing that it's utf8 is not the right thing to do. The encode (or is it decode? I always forget) defaults to utf8, which is what you should be using.
    - Siecje 9y ago
      It helps me to think encrypt for encode (convert to bytes) and decrypt for decode (convert to text).
  - viraptor 9y ago
    I'm not sure why this example is interesting. You can write: print('I DONT WANT TO add "UTF-8" everywhere!!') And get the same result. Or if you really want to start with bytes: print(b'some string'.decode())

Yes, please make UTF-8 default! It's 2018 print(str(b'I DONT WANT TO add "UTF-8" everywhere!!','UTF-8'))

As a serialization and transmission format, UTF-8 is fine. As the basis of a high-level language's string type, UTF-8 is objectively incorrect. Strings in a high-level language should be as clean an abstraction of Unicode as possible, and leaking implementation details of the particular byte-encoding scheme up to the programmer is not acceptable.

singularity2001 9y ago
Ok, is it reasonable to ask for all string getter/setter file read/write ops to default to UTF-8? also having different string methods .byte_length .char_length(?) .codepoint_length seems a good idea. or string.length(aspect=bytes) but what should be the default aspect?
- mjevans 9y ago
  For compatibility reasons, len(unicode type) should probably return the size if accessed as a raw 8 bit sequence, so an alias to .length_bytes. I also think the other names should similarly start with .length_ .
  - mark-r 9y ago
    len() should return the length in individual addressable units, so that obj[len(obj)-1] always returns the last item, whether that's a byte or a unicode character.
- ubernostrum 9y ago
  I'm going to quote something I wrote a couple years ago[1]: Now, I should point out here that I’m not really knocking the people who were writing, say, command-line and file-handling utilities in Python. For years, Python sort of accepted the status quo of the Unix world, which was mostly to stick its fingers in its ears and shout LA LA LA I CAN’T HEAR YOU I’M JUST GOING TO SET LC_CTYPE TO C AGAIN AND GO BACK TO MY HAPPY PLACE. A bit later on it changed to “just use UTF-8 everywhere, UTF-8 is perfectly safe”, which really meant “just use UTF-8 everywhere because we can continue pretending it’s ASCII up until the time someone gives us a non-ASCII or multi-byte character, at which point do the fingers-in-ears-can’t-hear-you thing again”. So a lot of what you’ll see in terms of complaints about string handling are really complaints that Unix’s pretend-everything-is-ASCII-until-it-breaks approach was never very good to begin with and just gets worse with every passing year. I stand by this: we had a couple of decades of Python catering to this brokenness, and it made life miserable for everyone who didn't work in that particular domain. Python 3 changed that. Does it mean life got harder for some people? Yup. But life got a lot easier and more reliable for many more people, and it's a tradeoff I'm willing to accept. [1] https://www.b-list.org/weblog/2016/jun/10/python-3-again/ https://www.b-list.org/weblog/2016/jun/10/python-3-again/
  - mjevans 9y ago
    Can you answer what ways things got easier that are /not/ equal or better in the solution I proposed?
    - viraptor 9y ago
      Currently in python you know that you're holding either raw bytes, or something that can be successfully serialised to utf8. With your proposed solution you'll find out which one it is when you try to encode it. It's the difference between easier to debug: "you asked me to read a value here, but your assumptions about the encoding don't match reality" exception and the hard to debug: "I did a lot of processing; you thought this thing is a valid text, but it isn't; have fun tracking down how it got here in the first place" exception.
      - mjevans 9y ago
        Python 3: We have two things that are ALMOST the same, and which if we'd done it correctly, could have been converted just by changing what we're willing to call it (in the "downgrade" direction; or also verifying if you want to achieve what I hate that Python 3 is forcing on programmers). Proposed "string like" Object: __IF__ you want to turn on debugging, sure, force it to validate assumptions at runtime/compile time. Otherwise call verify() when you're willing to handle that being some result indicating "we have a problem". Maybe the verify() call returns the byte-access-offset of the first non-conforming sequence.
        viraptor 9y ago
        > We have two things that are ALMOST the same, I think we've got a fundamental disagreement here. I don't believe they're similar at all. They just happened to get confused a lot in the past when it didn't matter that much.
    - ubernostrum 9y ago
      Which "solution" was that? Your comment suggesting that len() should be the length in bytes of a UTF-8 representation of the string? That would be a terrible thing for people who work with strings as strings, and would essentially tell such people -- who make up a lot of Python's user base! -- "go use something else, this language is only for sysadmins to write Unix utilities now".
- mark-r 9y ago
  Unfortunately defaulting to UTF-8 is often a bad choice in Windows. Curse Microsoft!
mjevans 9y ago
What's unacceptable is forcing the programmer to deal with the possibility of 'garbage' strings by default. Also having major issues when half of the libraries want 'binary string like sequences' and the other half want 'native Unicode strings'. That's what the support for a generic array type, which MIGHT be handled like a string (if the programmer chooses), and which when designated as being in a Unicode format of ANY type, and processed with data of any valid Unicode type, will automatically coerce together the correct way, and produce a valid result* (assuming valid inputs, and which might still need to actually be validated (again, bitflag cleared), if anyone cares). THAT is what a high level language should be. Python has actually __REGRESSED__ in that respect. It's handing of Unicode is //low// level. It forces the programmer to need to care about the data types, and deal with legacy invalid data strings in some method that's up to them. Except they can't /ignore/ invalid sequences, because that option isn't supported.

viraptor 9y ago

Not sure what you mean. It's the default for a long time now. https://docs.python.org/3/library/stdtypes.html#bytes.decode https://docs.python.org/3/library/stdtypes.html#bytes.decode `open` also defaults to your system encoding, which is likely utf8 judging from the message.

singularity2001 9y ago
print(str(b'I am a broken string')) print(str(b'I DONT WANT TO add "UTF-8" everywhere!!','UTF-8')) not quite default!
- orf 9y ago
  Explicit is better than implicit. The first line isnt a string, it's a collection of bytes, guessing that it's utf8 is not the right thing to do. The encode (or is it decode? I always forget) defaults to utf8, which is what you should be using.
  - Siecje 9y ago
    It helps me to think encrypt for encode (convert to bytes) and decrypt for decode (convert to text).
- viraptor 9y ago
  I'm not sure why this example is interesting. You can write: print('I DONT WANT TO add "UTF-8" everywhere!!') And get the same result. Or if you really want to start with bytes: print(b'some string'.decode())