5 ms·

> Having had to deal with internationalization before, I can honestly say that there's only one way to handle it: convert all string data to your internal repre

> Having had to deal with internationalization before, I can honestly say that there's only one way to handle it: convert all string data to your internal representation (be it UTF-8 (Rust), UTF-16 (Java), or UTF-32 (Python)) at the margin where you read/write it I'm not going to go into the actual subtleties here but there is a lot more to it than what you suggest here. However the biggest mistake you made here is to equate that any encoding is equivalent here. There is a tremendous difference between using utf-8 as internal encoding or a variable encoding like in Python 3. Rust and Go have free bytes/string conversion wheres Python 3 has none of that.

jcranmer 10y ago
Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32. The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped.
- the_mitsuhiko 10y ago
  > Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32. That is incorrect. The API assumes the string is O(1) indexable. It makes no assumptions or guarantees about the internal encoding. > The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped. This is an incorrect blanket statement and I can give you plenty of examples where this is exactly what you will have to do because the knowledge of the encoding is not available until later. This is in fact a common problem that Python 3 has because it decodes to unicode so early that it had to come up with surrogate-escapes to roundtrip bytes in case the encoding it guessed/inferred was wrong.
  - mattashii 10y ago
    He's talking about CPython, the default interpreter/compiler for Python in many systems. CPython internally uses 1-byte representation if the maximum Unicode code point of its chars is 127 or less (therefore ASCII), 2-byte representation for values less than 32767 (thus utf-16, but only the first half-page), and for the rest it uses UTF-32, which means all lookups can be done in O(1). Assuming UTF-8 encoding for source files is not that strange, as it is getting more and more the default or preferred encoding for string-based I/O.
    - the_mitsuhiko 10y ago
      I understand how CPython works however the external API makes no guarantees about any internal encoding. The implementation of the storage also changed many times.
echlebek 10y ago
Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning? I really like the way Go's strings work. I wish Python's strings worked like this too. https://play.golang.org/p/a70FEDOxBz https://play.golang.org/p/a70FEDOxBz
- the_mitsuhiko 10y ago
  > Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning? You can access bytes in go out of the string without even having to perform a conversion. For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy. (You will need to verify utf-8 integrity of course)
  - uluyol 10y ago
    Go strings can contain arbitrary binary data, they are not limited to valid utf-8.
  - echlebek 10y ago
    Oh I see what you mean now. > For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy. Super unholy :)

jcranmer 10y ago

Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32. The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped.

the_mitsuhiko 10y ago
> Python 3's string types are internally ASCII (or Latin-1, I'm not sure), UTF-16, or UTF-32 depending on the contents of the string, but the API effectively assumes that the encoding is UTF-32. That is incorrect. The API assumes the string is O(1) indexable. It makes no assumptions or guarantees about the internal encoding. > The main point is that you don't want to be passing around binary data internally and expecting people to decode charsets as needed--the knowledge of charsets will usually be dropped. This is an incorrect blanket statement and I can give you plenty of examples where this is exactly what you will have to do because the knowledge of the encoding is not available until later. This is in fact a common problem that Python 3 has because it decodes to unicode so early that it had to come up with surrogate-escapes to roundtrip bytes in case the encoding it guessed/inferred was wrong.
- mattashii 10y ago
  He's talking about CPython, the default interpreter/compiler for Python in many systems. CPython internally uses 1-byte representation if the maximum Unicode code point of its chars is 127 or less (therefore ASCII), 2-byte representation for values less than 32767 (thus utf-16, but only the first half-page), and for the rest it uses UTF-32, which means all lookups can be done in O(1). Assuming UTF-8 encoding for source files is not that strange, as it is getting more and more the default or preferred encoding for string-based I/O.
  - the_mitsuhiko 10y ago
    I understand how CPython works however the external API makes no guarantees about any internal encoding. The implementation of the storage also changed many times.

echlebek 10y ago

Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning? I really like the way Go's strings work. I wish Python's strings worked like this too. https://play.golang.org/p/a70FEDOxBz https://play.golang.org/p/a70FEDOxBz

the_mitsuhiko 10y ago
> Do you have a source for Go having free byte/string conversion? To my knowledge it is a copy. Or maybe I don't get your meaning? You can access bytes in go out of the string without even having to perform a conversion. For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy. (You will need to verify utf-8 integrity of course)
- uluyol 10y ago
  Go strings can contain arbitrary binary data, they are not limited to valid utf-8.
- echlebek 10y ago
  Oh I see what you mean now. > For bytes to string you can use the unsafe and reflect package to create strings from bytes without a copy. Super unholy :)