5 ms·

All the standard libraries need to use it too. golang gets this part right. In golang strings are immutable sequences of bytes (which can be concatenated) or

All the standard libraries need to use it too. golang gets this part right. In golang strings are immutable sequences of bytes (which can be concatenated) or have 'slices' taken from within them (I believe addressed to a rune; which IIRC is a complete UTF-8 sequence). Standard libraries typically operate with bytes[] for low level operations (file I/O) but strings and bytes can be coerced between each other with ease, and notably many objects (such as a File https://golang.org/pkg/os/#File.Write https://golang.org/pkg/os/#File.Write ) have methods for interacting with bytes[] and strings. There's a core library specifically for low level Unicode operations and external libraries for more complex tasks.

int_19h 7y ago
Go gets this very wrong, because you can still index strings as arrays - and you get bytes back, which 99% of the time is not what you want. But which also happens to "just work" on common ASCII inputs, so you get code floating around that is broken without its authors' knowledge. (This, coincidentally, was also the problem in Python 2, and why Python 3 breaking the world was necessary, if inconvenient for some.) At this point, the only sensible design for strings is as fully encapsulated, opaque values that are emphatically not arrays of anything. If you want to index them, you should be specifically saying what you want - nth byte, nth codepoint, nth grapheme etc. If you want content as an array of bytes in a particular encoding, then you should ask for that specifically. Internally, yeah, it should use UTF-8, because it makes the most sense - but that's just an implementation detail. So far, the only language I know that got this right is Swift.
- adwn 7y ago
  Rust also gets this right: Every string and every string slice is guaranteed to be a valid UTF-8 sequence. If you slice a string at invalid UTF-8 boundaries, then – depending on the operation – the thread either panics, or None is returned.
- alexeldeib 7y ago
  I agree with your stated goals, but I'm less convinced Go gets this totally wrong. It has the nice property that a string is a string is a string -- a slice of bytes. The only exception is raw strings which as part of source code are defined to be UTF-8. So, while there's a tradeoff in elegance I do think it's logically very simple. I do think the Rust approach for example is more "correct", but it's also tedious to work with and I think the ergonomics matter (can't speak for swift).
- mjevans 7y ago
  How do you feel about how golang handles runes? Example: https://play.golang.org/p/7XWC9Irui3N https://play.golang.org/p/7XWC9Irui3N x = abc 3 vs 3 y = é 3 vs 2 z = 世界 6 vs 2 s>>r 0 1 true e s>>r 1 2 true ́ []b>>b 0 true true 00000065 []b>>b 1 false true 000000CC []b>>b 2 true false 00000081 (not putting in a code block given the above and mobile...) package main import ( "fmt" "unicode/utf8" ) func main() { var x, y, z string; x = "abc"; y = string([]byte{0x65, 0xCC, 0x81}) // 'e' + 0x0301 ( 0(011 00)(00 0001)) = "é"; z = "世界"; fmt.Printf("x = %s\n%d vs %d\n\ny = %s\n%d vs %d\n\nz = %s\n%d vs %d\n\n", x, len(x), utf8.RuneCountInString(x), y, len(y), utf8.RuneCountInString(y), z, len(z), utf8.RuneCountInString(y)); for k, v := range y { // iterates as runes, not string slices //fmt.Printf("s %d\t%s\t%v\t%s\n", k, utf8.RuneCountInString(v), utf8.ValidString(v), v); fmt.Printf("s>>r %d\t%d\t%v\t%c\n", k, utf8.RuneLen(v), utf8.ValidRune(v), v); } for k, v := range []byte(y) { // iterates as byte units, not runes fmt.Printf("[]b>>b %d\t%v\t%v\t%08X\n", k, utf8.FullRune([]byte{v}), utf8.RuneStart(v), v); } }
bonoboTP 7y ago
It seems like you want to do string-like standard library operations on byte arrays. Python 3 is clear here. You can do string stuff with strings only. If you want to do string stuff, you must decode the bytes. The world is not only about English and ASCII. People who want these shortcuts usually don't understand what problems Unicode and character encodings solve in general. And that's exactly what produces software that won't allow me to properly enter my Hungarian name as it is, with diacritics. It's utterly inexcusable and lazy and stems from an annoying sense of Anglo superiority.

int_19h 7y ago

Go gets this very wrong, because you can still index strings as arrays - and you get bytes back, which 99% of the time is not what you want. But which also happens to "just work" on common ASCII inputs, so you get code floating around that is broken without its authors' knowledge. (This, coincidentally, was also the problem in Python 2, and why Python 3 breaking the world was necessary, if inconvenient for some.) At this point, the only sensible design for strings is as fully encapsulated, opaque values that are emphatically not arrays of anything. If you want to index them, you should be specifically saying what you want - nth byte, nth codepoint, nth grapheme etc. If you want content as an array of bytes in a particular encoding, then you should ask for that specifically. Internally, yeah, it should use UTF-8, because it makes the most sense - but that's just an implementation detail. So far, the only language I know that got this right is Swift.

adwn 7y ago
Rust also gets this right: Every string and every string slice is guaranteed to be a valid UTF-8 sequence. If you slice a string at invalid UTF-8 boundaries, then – depending on the operation – the thread either panics, or None is returned.
alexeldeib 7y ago
I agree with your stated goals, but I'm less convinced Go gets this totally wrong. It has the nice property that a string is a string is a string -- a slice of bytes. The only exception is raw strings which as part of source code are defined to be UTF-8. So, while there's a tradeoff in elegance I do think it's logically very simple. I do think the Rust approach for example is more "correct", but it's also tedious to work with and I think the ergonomics matter (can't speak for swift).
mjevans 7y ago
How do you feel about how golang handles runes? Example: https://play.golang.org/p/7XWC9Irui3N https://play.golang.org/p/7XWC9Irui3N x = abc 3 vs 3 y = é 3 vs 2 z = 世界 6 vs 2 s>>r 0 1 true e s>>r 1 2 true ́ []b>>b 0 true true 00000065 []b>>b 1 false true 000000CC []b>>b 2 true false 00000081 (not putting in a code block given the above and mobile...) package main import ( "fmt" "unicode/utf8" ) func main() { var x, y, z string; x = "abc"; y = string([]byte{0x65, 0xCC, 0x81}) // 'e' + 0x0301 ( 0(011 00)(00 0001)) = "é"; z = "世界"; fmt.Printf("x = %s\n%d vs %d\n\ny = %s\n%d vs %d\n\nz = %s\n%d vs %d\n\n", x, len(x), utf8.RuneCountInString(x), y, len(y), utf8.RuneCountInString(y), z, len(z), utf8.RuneCountInString(y)); for k, v := range y { // iterates as runes, not string slices //fmt.Printf("s %d\t%s\t%v\t%s\n", k, utf8.RuneCountInString(v), utf8.ValidString(v), v); fmt.Printf("s>>r %d\t%d\t%v\t%c\n", k, utf8.RuneLen(v), utf8.ValidRune(v), v); } for k, v := range []byte(y) { // iterates as byte units, not runes fmt.Printf("[]b>>b %d\t%v\t%v\t%08X\n", k, utf8.FullRune([]byte{v}), utf8.RuneStart(v), v); } }

bonoboTP 7y ago

It seems like you want to do string-like standard library operations on byte arrays. Python 3 is clear here. You can do string stuff with strings only. If you want to do string stuff, you must decode the bytes. The world is not only about English and ASCII. People who want these shortcuts usually don't understand what problems Unicode and character encodings solve in general. And that's exactly what produces software that won't allow me to properly enter my Hungarian name as it is, with diacritics. It's utterly inexcusable and lazy and stems from an annoying sense of Anglo superiority.