4 ms·

I'm in similar situation wrt what I use python for, and suffered from the same apathy until quite recently. Man, it was really easy to switch, and completely wo

I'm in similar situation wrt what I use python for, and suffered from the same apathy until quite recently. Man, it was really easy to switch, and completely worth it just for the little things (like UTF-8 as default encoding for strings). It's really painless, but if you want to ease into it, I recommend using __future__ in your py2 stuff for now just to get you into a py3 state of mind: https://docs.python.org/2/library/__future__.html https://docs.python.org/2/library/__future__.html

deleted 9y ago
[deleted]
Animats 9y ago
UTF-8 isn't CPython's default encoding for strings. The internal encodings are ASCII, UTF-16, or UTF-32, depending on the widest character in the string. I would have made UTF-8 the internal representation, and generated an array of subscript indices only when someone random-accessed a string. If the string is being processed sequentially, as with for c in s : ... you don't need an index array. You don't need them for list comprehensions or regular expressions. One could even have opaque string indices, returned by search methods, which don't require an index array unless forcibly converted to an integer. Some special cases, such as "s[-1]", don't really need an index array either.
- collinmanderson 9y ago
  I agree. Go does something similar (using UTF-8). The Go creators _invented_ UTF-8, so I trust them. :)
  - Animats 9y ago
    Go uses UTF-8 internally, but you index a Go string by bytes, not glyphs or graphemes. The "index" function on strings in Go returns an integer, not an opaque index object or a slice. You can create a slice of a UTF-8 string which is misaligned and not valid UTF-8. Indexing a string in Python 3 returns glyphs (which are strings), not bytes. Not graphemes; if you index through a Python 3 string with emoji that have skin color modifications, you'll get the emoji glyph and the skin color modifier as separate items. Python 3 also has a type "bytes", which can't quite decide whether it's a string type or an array of integers. Print a "bytes" type, and it's printed as a string, with a 'b" in front of it, not as an array of integers. But an element of a "bytes" array is an int, not a bytes type. This is for backwards compatibility; it's roughly compatible with legacy "str" from Python 2. Rust struggles with this. Rust tries to prevent you from getting a invalid UTF-8 string. The solution used involves re-checking for UTF-8 validity a lot, or bypassing it with unsafe code. This gets messy.
    - steveklabnik 9y ago
      > The solution used involves re-checking for UTF-8 validity a lot, You only need to check once, at the boundary of when you're converting from something that may or may not be UTF-8 to a UTF-8 string.

deleted 9y ago

[deleted]

Animats 9y ago

UTF-8 isn't CPython's default encoding for strings. The internal encodings are ASCII, UTF-16, or UTF-32, depending on the widest character in the string. I would have made UTF-8 the internal representation, and generated an array of subscript indices only when someone random-accessed a string. If the string is being processed sequentially, as with for c in s : ... you don't need an index array. You don't need them for list comprehensions or regular expressions. One could even have opaque string indices, returned by search methods, which don't require an index array unless forcibly converted to an integer. Some special cases, such as "s[-1]", don't really need an index array either.

collinmanderson 9y ago
I agree. Go does something similar (using UTF-8). The Go creators _invented_ UTF-8, so I trust them. :)
- Animats 9y ago
  Go uses UTF-8 internally, but you index a Go string by bytes, not glyphs or graphemes. The "index" function on strings in Go returns an integer, not an opaque index object or a slice. You can create a slice of a UTF-8 string which is misaligned and not valid UTF-8. Indexing a string in Python 3 returns glyphs (which are strings), not bytes. Not graphemes; if you index through a Python 3 string with emoji that have skin color modifications, you'll get the emoji glyph and the skin color modifier as separate items. Python 3 also has a type "bytes", which can't quite decide whether it's a string type or an array of integers. Print a "bytes" type, and it's printed as a string, with a 'b" in front of it, not as an array of integers. But an element of a "bytes" array is an int, not a bytes type. This is for backwards compatibility; it's roughly compatible with legacy "str" from Python 2. Rust struggles with this. Rust tries to prevent you from getting a invalid UTF-8 string. The solution used involves re-checking for UTF-8 validity a lot, or bypassing it with unsafe code. This gets messy.
  - steveklabnik 9y ago
    > The solution used involves re-checking for UTF-8 validity a lot, You only need to check once, at the boundary of when you're converting from something that may or may not be UTF-8 to a UTF-8 string.