5 ms·

I've written commercial code to handle decidedly non-english languages in python2 just fine. Have you? Do you understand unicode and it's encodings well enough

by patrec 9y ago

I've written commercial code to handle decidedly non-english languages in python2 just fine. Have you? Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like.

svennek 9y ago
Yes, I have (for more than a decade in four different languages, not all of those who are latin1)... One project was even both LTR and RTL :) Byte handling is definately not fun in variable encodings, and even fixed-length multi-byte encodings require a lot of support functionality when working with bytes.. Such a simple thing as "len" is suddenly hard (for the human definition of length and not the byte-length), the latter is also easily attainable by converting the string to bytes and get the length of that in the given encoding.
- patrec 9y ago
  len isn't a great example because it doesn't do anything remotely useful on python3 strs in the general case. In particular it doesn't tell you how many characters the str has, which is what you probably wanted to know -- possibly as an approximation for figuring out how many columns in fixed-width it would take, in which case of course you are even more likely to be disappointed. In python2 len(str) will tell you exactly how many bytes some text takes, which is useful.
masklinn 9y ago
> Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like. * Rust has a very strict split between bytes and text * Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying * Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)
- dom0 9y ago
  > Erlang What the fuck. And I thought constant-time indexable strings were a bad idea!
  - masklinn 9y ago
    FWIW it's way too common in functional languages, Haskell's standard String is a typedef for [Char] (a linked list of Char) hence the proliferation of various other string types.
  - patrec 9y ago
    They are. You want to use iolists in Erlang. Constant time concatenation (which is useful) rather than constant time indexing into codepoints (which is not; and erlang does not have BTW).
- patrec 9y ago
  Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design and has very nice features that are pretty much unique to Erlang. Erlang has 3 types of strings: 1. the list of integers thing, which you should mostly avoid 2. binary strings (like python2 str or go) 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you? See http://www.evanmiller.org/elixir-ram-and-the-template-of-doom.html http://www.evanmiller.org/elixir-ram-and-the-template-of-doo... for more details.
  - masklinn 9y ago
    > Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful. > 2. binary strings (like python2 str or go) AKA not actually strings, I don't remember Erlang having much if any string-manipulation API for working on bytes. > 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. This is only for output, so they're not actually ropes. > So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you? Most mutable languages will use buffered output streams and possibly some sort of string sequence (even C has iovecs), so while iolists are a good fit for Erlang they make sense in few other languages.
ubernostrum 9y ago
Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? I do, and I do believe Python 3's approach is superior for strings. An array of 8-bit integers is not a string. It's an array of 8-bit integers. I want my string type to be a sequence of Unicode codepoints, and I want iterating it to yield codepoints, I want indexing into it to yield the codepoint at the index, and I want the length of it to be the number of codepoints it contains. When I'm ready to send it over a wire or flush it to disk, then I'll think about the best way to represent it as an array of 8-bit integers, but bringing the baggage of encodings (especially the baggage of UTF-8) into any part of my code except the boundary layer is asking for trouble.
- patrec 9y ago
  What's your use case for iterating or indexing into code points?
  - ubernostrum 9y ago
    My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes. I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
    - patrec 9y ago
      But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either a) low-level mucking around like normalization -- a fringe use b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.
rtpg 9y ago
I handled a transition of a multilingual/multiencoding Python 2 project to Python 3. Firstly, you can _still_ work entirely in bytes in Python 3 if you want to. But you can't call print on them... For good reason! You don't know the encoding to use to display them! When we transitioned, there were a lot of subtle issues in our code due to our almost cargo cult usage of encoded/decode that ended with a lot of redundant transformations. Loads of that disappeared afterwards. You can still operate in "Python 2 mode" in Python 3. The only exception I can think of is the difficulty in getting a bytestream version of standard output. Also... Python 2 is decidably not like Go or Rust. The simplest example is how json.dumps would return bytes or a Unicode string depending on the keys on your object. No respect for typing
- dom0 9y ago
  > The only exception I can think of is the difficulty in getting a bytestream version of standard output. sys.{stderr, stdout, stdin}.buffer

by patrec 9y ago

svennek 9y ago

Yes, I have (for more than a decade in four different languages, not all of those who are latin1)... One project was even both LTR and RTL :) Byte handling is definately not fun in variable encodings, and even fixed-length multi-byte encodings require a lot of support functionality when working with bytes.. Such a simple thing as "len" is suddenly hard (for the human definition of length and not the byte-length), the latter is also easily attainable by converting the string to bytes and get the length of that in the given encoding.

patrec 9y ago
len isn't a great example because it doesn't do anything remotely useful on python3 strs in the general case. In particular it doesn't tell you how many characters the str has, which is what you probably wanted to know -- possibly as an approximation for figuring out how many columns in fixed-width it would take, in which case of course you are even more likely to be disappointed. In python2 len(str) will tell you exactly how many bytes some text takes, which is useful.

masklinn 9y ago

> Hint: this is much closer to what string handling in sensibly designed languages like Rust and Go or Erlang looks like. * Rust has a very strict split between bytes and text * Go has fucked up bytestrings which depending on validity may break at any point depending on the API you're applying * Erlang's strings are widely considered awful, and are literally just linked lists of integers making them slow, memory-heavy and prone to pretty-printer whims (as the pretty printer decides on the fly whether a given list looks like a string — and should be printed thus — or does not — and should be printed as a regular list of integers)

dom0 9y ago
> Erlang What the fuck. And I thought constant-time indexable strings were a bad idea!
- masklinn 9y ago
  FWIW it's way too common in functional languages, Haskell's standard String is a typedef for [Char] (a linked list of Char) hence the proliferation of various other string types.
- patrec 9y ago
  They are. You want to use iolists in Erlang. Constant time concatenation (which is useful) rather than constant time indexing into codepoints (which is not; and erlang does not have BTW).
patrec 9y ago
Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design and has very nice features that are pretty much unique to Erlang. Erlang has 3 types of strings: 1. the list of integers thing, which you should mostly avoid 2. binary strings (like python2 str or go) 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you? See http://www.evanmiller.org/elixir-ram-and-the-template-of-doom.html http://www.evanmiller.org/elixir-ram-and-the-template-of-doo... for more details.
- masklinn 9y ago
  > Erlang's strings might be widely considered awful, but probably not by people who actually have a clue – the numbers thing is ugly, but overall it's a very nice design It is nice in a theoretical sense, in the same way Haskell's [Char] is nice, in a practical sense it's awful. > 2. binary strings (like python2 str or go) AKA not actually strings, I don't remember Erlang having much if any string-manipulation API for working on bytes. > 3. iolists: a list or element of either of 1, 2 or 3. This is the nice part – it basically means erlang has ropes. And all the "egress" functions (sending udp or tcp packets or writing to stdout or files or sockets etc.) take iolists. This is only for output, so they're not actually ropes. > So you can avoid a lot of pointless copying and write text or binary data very efficiently; I know of no other (non-BEAM) language that has similarly good support for this, do you? Most mutable languages will use buffered output streams and possibly some sort of string sequence (even C has iovecs), so while iolists are a good fit for Erlang they make sense in few other languages.

ubernostrum 9y ago

Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? I do, and I do believe Python 3's approach is superior for strings. An array of 8-bit integers is not a string. It's an array of 8-bit integers. I want my string type to be a sequence of Unicode codepoints, and I want iterating it to yield codepoints, I want indexing into it to yield the codepoint at the index, and I want the length of it to be the number of codepoints it contains. When I'm ready to send it over a wire or flush it to disk, then I'll think about the best way to represent it as an array of 8-bit integers, but bringing the baggage of encodings (especially the baggage of UTF-8) into any part of my code except the boundary layer is asking for trouble.

patrec 9y ago
What's your use case for iterating or indexing into code points?
- ubernostrum 9y ago
  My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes. I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
  - patrec 9y ago
    But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either a) low-level mucking around like normalization -- a fringe use b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.

rtpg 9y ago

I handled a transition of a multilingual/multiencoding Python 2 project to Python 3. Firstly, you can _still_ work entirely in bytes in Python 3 if you want to. But you can't call print on them... For good reason! You don't know the encoding to use to display them! When we transitioned, there were a lot of subtle issues in our code due to our almost cargo cult usage of encoded/decode that ended with a lot of redundant transformations. Loads of that disappeared afterwards. You can still operate in "Python 2 mode" in Python 3. The only exception I can think of is the difficulty in getting a bytestream version of standard output. Also... Python 2 is decidably not like Go or Rust. The simplest example is how json.dumps would return bytes or a Unicode string depending on the keys on your object. No respect for typing

dom0 9y ago
> The only exception I can think of is the difficulty in getting a bytestream version of standard output. sys.{stderr, stdout, stdin}.buffer