3 ms·

I agree. Go does something similar (using UTF-8). The Go creators _invented_ UTF-8, so I trust them. :)

• Animats 9y ago
  Go uses UTF-8 internally, but you index a Go string by bytes, not glyphs or graphemes. The "index" function on strings in Go returns an integer, not an opaque index object or a slice. You can create a slice of a UTF-8 string which is misaligned and not valid UTF-8. Indexing a string in Python 3 returns glyphs (which are strings), not bytes. Not graphemes; if you index through a Python 3 string with emoji that have skin color modifications, you'll get the emoji glyph and the skin color modifier as separate items. Python 3 also has a type "bytes", which can't quite decide whether it's a string type or an array of integers. Print a "bytes" type, and it's printed as a string, with a 'b" in front of it, not as an array of integers. But an element of a "bytes" array is an int, not a bytes type. This is for backwards compatibility; it's roughly compatible with legacy "str" from Python 2. Rust struggles with this. Rust tries to prevent you from getting a invalid UTF-8 string. The solution used involves re-checking for UTF-8 validity a lot, or bypassing it with unsafe code. This gets messy.
  • steveklabnik 9y ago
    > The solution used involves re-checking for UTF-8 validity a lot, You only need to check once, at the boundary of when you're converting from something that may or may not be UTF-8 to a UTF-8 string.