4 ms·

Coming from C# it seems odd that a string would index on byte and not char(rune) and that it is essentially a read only byte array. If you wanted a byte array w

Coming from C# it seems odd that a string would index on byte and not char(rune) and that it is essentially a read only byte array. If you wanted a byte array why wouldn't you use a byte array, why have strings and byte arrays? In C# you can encode/decode strings to byte arrays based on your desired encoding, but a string is composed of characters, it's in memory representation is abstracted. Is this a performance or zero copy thing? Not having to encode/decode to get to the bytes?

dsymonds 13y ago
The string type is immutable, which is the main reason for using it instead of []byte.
- SigmundA 13y ago
  Same in C#, but why not have an immutable byte array for bytes and a string for char/runes? Honestly never needed an immutable byte array that wasn't for chars.
  - dsymonds 13y ago
    Go doesn't have the concept of an immutable array/slice. It's been proposed (largely to bridge the divide between string and []byte), but it doesn't fit quite right with the rest of the language.
  - twotwotwo 13y ago
    Conversely, I could say why not use bytes for both, and that when I need an index lookup it's usually a byte offset. But I think neither statement is totally satisfying, and the more interesting question is why different languages settled on different encodings for strings in RAM. Microsoft had reasons to pick UTF-16 for C# and the CLR. The Windows API speaks UTF-16, and when C# was first announced way back in 2000, UTF-8 was not yet a widely used encoding on the Web; Unicode was sadly not that widely used on the Web, period. The decision to use 16-bit Unicode in the Win32 API went even further back, to the development leading up to NT 3.1's release in July 1993. At that point, UTF-8 was a relative baby; it was presented at USENIX in January 1993. Also, back then, code points basically were 16 bits because surrogate pairs were but a twinkle in the Unicode Consortium's eye. The Unicode 2.0 standard added surrogate pairs to help them expand their CJK selection and generally let them add more chars of all sorts; it wasn't released until 1996. Now here we are and we have 😃, U+1F603. Go came along in 2009; by then, many Web sites were being served in UTF-8, and Go'd be used in significant part in Web operations, and UTF-8 was also the default encoding in many Unix environments. Go initially didn't run on Windows at all and was ported by the community, so fitting in with Win32 wasn't an issue. Code points that wouldn't fit in 16 bits were a fact of life by '09, too. Arguments about inherent merits of encodings aside, it probably would have seemed to lots of folks that UTF-8 was a natural choice for that task at that time. Also, Pike and Thompson are two of the three co-designers of Go and co-designed UTF-8, and UTF-8 was the encoding used by the Plan 9 OS/environment they built at Bell Labs, so, again, technical details aside, it was kind of a foregone conclusion which encoding they'd build the language around. :) The one thing I do not want to do here is get in an argument about the inherent merits of character encodings, so I'm just not gonna do that. :) On UTF-16 and Windows NT: http://support.microsoft.com/kb/99884 http://support.microsoft.com/kb/99884, http://en.wikipedia.org/wiki/Windows_NT_3.1 http://en.wikipedia.org/wiki/Windows_NT_3.1, and http://en.wikipedia.org/wiki/Unicode http://en.wikipedia.org/wiki/Unicode On Unicode adoption on the Web, UTF-8, and Plan 9: http://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html http://googleblog.blogspot.com/2012/02/unicode-over-60-perce..., http://en.wikipedia.org/wiki/UTF-8 http://en.wikipedia.org/wiki/UTF-8, http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
porges 13y ago
Note that in C# indexing on char is really indexing on UTF-16 codepoint, which is arguably worse because it seems to work, until it doesn't.
- SigmundA 13y ago
  Interesting, never had to use use StringInfo to get into surrogate pair support. Although its seems like this could be fixed since both char and string are abstracted from bytes, although indexing on variable length chars would increase computation or UTF-32 would increase memory.
skybrian 13y ago
Well, the first question is what a string's internal representation should be, and they went with UTF8. Once you've decided on UTF8, the question is whether to hide the representation or expose it. If you decide to expose it, there's hardly any difference between an immutable byte array and string, so it's simpler in a way to have one type that can be used both ways.
- dsymonds 13y ago
  No, read the article more carefully. A Go string's internal representation is a sequence of bytes. There's nothing UTF-8 about it.
  - skybrian 13y ago
    I understood the article just fine. It's not wrong to say that in Go, text (called strings in other languages) is nearly always represented in UTF8 format and is commonly stored in variables of type string, or sometimes []byte. Perhaps I should have left out the word "internal" since it's exposed.
    - dsymonds 13y ago
      Yes, "internal representation" has a specific meaning. If you had said "standard representation" or something then I would have agreed with you.
  - dchest 13y ago
    Yes, there is: range on a string works on runes by decoding UTF-8.
    - dsymonds 13y ago
      That has nothing to do with "a string's internal representation". That is just how a range loop is defined/implemented.
bazzargh 13y ago
As someone's already pointed out, C# strings are composed of UTF-16 codepoints not characters - this means that if you have a character outside the basic multilingual plane it'll be represented as two codepoints using a surrogate, and the character count in the C# string will be wrong (the same is true of Java and JS for example) That's a hard problem, and avoiding it in every situation would require scanning the strings for surrogates beforehand, when you might never need to know that information. Go makes it explicit that knowing the exact character position and string length in characters comes at a cost. There's a good discussion of this on Tim Bray's blog: http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF
- twotwotwo 13y ago
  Just for fun, here's Go handling a char outside the BMP (😃, U+1F603): http://play.golang.org/p/qg7POYAAOL http://play.golang.org/p/qg7POYAAOL
  - samatman 13y ago
    https://github.com/mnemnion/emojure/ https://github.com/mnemnion/emojure/ You can even export them without Capital letters ;-)
    - twotwotwo 13y ago
      I did run into at least one eminently reasonable use of how Go source is defined to be in UTF-8. Comments in the crypto libs just use math symbols where they're handy, like this in crypto/rsa[1]: // Check that de ≡ 1 mod p-1, for each prime. // This implies that e is coprime to each p-1 as e has a multiplicative // inverse. Therefore e is coprime to lcm(p-1,q-1,r-1,...) = // exponent(ℤ/nℤ). It also implies that a^de ≡ a mod p as a^(p-1) ≡ 1 // mod p. Thus a^de ≡ a mod n for all a coprime to n, as required. Sadly, the spec requires identifiers to be just Unicode letters and digits, so we will never experience the power and glory of emoji function names in Go. [1] http://golang.org/src/pkg/crypto/rsa/rsa.go http://golang.org/src/pkg/crypto/rsa/rsa.go
neild 13y ago
Indexing on runes is expensive, since either you need to store strings as arrays of runes or iterate on each index operation. Indexing on runes is also less useful than it may first seem. Consider that a rune is distinct from a glyph--a single "character" on the screen may be composed of several runes. The occasions when you care about specific runes as opposed to substrings are uncommon. When dealing with substrings, there is no advantage to substring-of-runes as opposed to substring-of-bytes. Note that many languages that appear to offer indexing by rune (e.g., Java) do not in fact do so, since their 16-bit "character" type is incapable of representing all runes. The fact that this is only rarely an issue points at the fundamental rarity with with code needs to deal with runes-qua-runes.
johnesj33 13y ago
UTF-8 strings ARE byte arrays -- just with some extra semantics for certain operations. Many operations don't even need to be aware of anything besides the raw array of bytes. C# has the worst-of-all-worlds approach and seems to cling to the specious notion that you can just "index" the "characters". In reality, you can't reliably index without the risk of splitting surrogate pairs and the notion that UTF-16 is an "array of characters" is completely false at all levels. The memory representation may be "abstracted", but it's not even a leaky abstraction, it's a historical accident and a complete broken abstraction. Beware of believing what Microsoft say about C# string handling -- they're stuck with UTF-16, whether they like it or not.

dsymonds 13y ago

The string type is immutable, which is the main reason for using it instead of []byte.

SigmundA 13y ago
Same in C#, but why not have an immutable byte array for bytes and a string for char/runes? Honestly never needed an immutable byte array that wasn't for chars.
- dsymonds 13y ago
  Go doesn't have the concept of an immutable array/slice. It's been proposed (largely to bridge the divide between string and []byte), but it doesn't fit quite right with the rest of the language.
- twotwotwo 13y ago
  Conversely, I could say why not use bytes for both, and that when I need an index lookup it's usually a byte offset. But I think neither statement is totally satisfying, and the more interesting question is why different languages settled on different encodings for strings in RAM. Microsoft had reasons to pick UTF-16 for C# and the CLR. The Windows API speaks UTF-16, and when C# was first announced way back in 2000, UTF-8 was not yet a widely used encoding on the Web; Unicode was sadly not that widely used on the Web, period. The decision to use 16-bit Unicode in the Win32 API went even further back, to the development leading up to NT 3.1's release in July 1993. At that point, UTF-8 was a relative baby; it was presented at USENIX in January 1993. Also, back then, code points basically were 16 bits because surrogate pairs were but a twinkle in the Unicode Consortium's eye. The Unicode 2.0 standard added surrogate pairs to help them expand their CJK selection and generally let them add more chars of all sorts; it wasn't released until 1996. Now here we are and we have 😃, U+1F603. Go came along in 2009; by then, many Web sites were being served in UTF-8, and Go'd be used in significant part in Web operations, and UTF-8 was also the default encoding in many Unix environments. Go initially didn't run on Windows at all and was ported by the community, so fitting in with Win32 wasn't an issue. Code points that wouldn't fit in 16 bits were a fact of life by '09, too. Arguments about inherent merits of encodings aside, it probably would have seemed to lots of folks that UTF-8 was a natural choice for that task at that time. Also, Pike and Thompson are two of the three co-designers of Go and co-designed UTF-8, and UTF-8 was the encoding used by the Plan 9 OS/environment they built at Bell Labs, so, again, technical details aside, it was kind of a foregone conclusion which encoding they'd build the language around. :) The one thing I do not want to do here is get in an argument about the inherent merits of character encodings, so I'm just not gonna do that. :) On UTF-16 and Windows NT: http://support.microsoft.com/kb/99884 http://support.microsoft.com/kb/99884, http://en.wikipedia.org/wiki/Windows_NT_3.1 http://en.wikipedia.org/wiki/Windows_NT_3.1, and http://en.wikipedia.org/wiki/Unicode http://en.wikipedia.org/wiki/Unicode On Unicode adoption on the Web, UTF-8, and Plan 9: http://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html http://googleblog.blogspot.com/2012/02/unicode-over-60-perce..., http://en.wikipedia.org/wiki/UTF-8 http://en.wikipedia.org/wiki/UTF-8, http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs

porges 13y ago

Note that in C# indexing on char is really indexing on UTF-16 codepoint, which is arguably worse because it seems to work, until it doesn't.

SigmundA 13y ago
Interesting, never had to use use StringInfo to get into surrogate pair support. Although its seems like this could be fixed since both char and string are abstracted from bytes, although indexing on variable length chars would increase computation or UTF-32 would increase memory.

skybrian 13y ago

Well, the first question is what a string's internal representation should be, and they went with UTF8. Once you've decided on UTF8, the question is whether to hide the representation or expose it. If you decide to expose it, there's hardly any difference between an immutable byte array and string, so it's simpler in a way to have one type that can be used both ways.

dsymonds 13y ago
No, read the article more carefully. A Go string's internal representation is a sequence of bytes. There's nothing UTF-8 about it.
- skybrian 13y ago
  I understood the article just fine. It's not wrong to say that in Go, text (called strings in other languages) is nearly always represented in UTF8 format and is commonly stored in variables of type string, or sometimes []byte. Perhaps I should have left out the word "internal" since it's exposed.
  - dsymonds 13y ago
    Yes, "internal representation" has a specific meaning. If you had said "standard representation" or something then I would have agreed with you.
- dchest 13y ago
  Yes, there is: range on a string works on runes by decoding UTF-8.
  - dsymonds 13y ago
    That has nothing to do with "a string's internal representation". That is just how a range loop is defined/implemented.

bazzargh 13y ago

As someone's already pointed out, C# strings are composed of UTF-16 codepoints not characters - this means that if you have a character outside the basic multilingual plane it'll be represented as two codepoints using a surrogate, and the character count in the C# string will be wrong (the same is true of Java and JS for example) That's a hard problem, and avoiding it in every situation would require scanning the strings for surrogates beforehand, when you might never need to know that information. Go makes it explicit that knowing the exact character position and string length in characters comes at a cost. There's a good discussion of this on Tim Bray's blog: http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF

twotwotwo 13y ago
Just for fun, here's Go handling a char outside the BMP (😃, U+1F603): http://play.golang.org/p/qg7POYAAOL http://play.golang.org/p/qg7POYAAOL
- samatman 13y ago
  https://github.com/mnemnion/emojure/ https://github.com/mnemnion/emojure/ You can even export them without Capital letters ;-)
  - twotwotwo 13y ago
    I did run into at least one eminently reasonable use of how Go source is defined to be in UTF-8. Comments in the crypto libs just use math symbols where they're handy, like this in crypto/rsa[1]: // Check that de ≡ 1 mod p-1, for each prime. // This implies that e is coprime to each p-1 as e has a multiplicative // inverse. Therefore e is coprime to lcm(p-1,q-1,r-1,...) = // exponent(ℤ/nℤ). It also implies that a^de ≡ a mod p as a^(p-1) ≡ 1 // mod p. Thus a^de ≡ a mod n for all a coprime to n, as required. Sadly, the spec requires identifiers to be just Unicode letters and digits, so we will never experience the power and glory of emoji function names in Go. [1] http://golang.org/src/pkg/crypto/rsa/rsa.go http://golang.org/src/pkg/crypto/rsa/rsa.go

neild 13y ago

Indexing on runes is expensive, since either you need to store strings as arrays of runes or iterate on each index operation. Indexing on runes is also less useful than it may first seem. Consider that a rune is distinct from a glyph--a single "character" on the screen may be composed of several runes. The occasions when you care about specific runes as opposed to substrings are uncommon. When dealing with substrings, there is no advantage to substring-of-runes as opposed to substring-of-bytes. Note that many languages that appear to offer indexing by rune (e.g., Java) do not in fact do so, since their 16-bit "character" type is incapable of representing all runes. The fact that this is only rarely an issue points at the fundamental rarity with with code needs to deal with runes-qua-runes.

johnesj33 13y ago

UTF-8 strings ARE byte arrays -- just with some extra semantics for certain operations. Many operations don't even need to be aware of anything besides the raw array of bytes. C# has the worst-of-all-worlds approach and seems to cling to the specious notion that you can just "index" the "characters". In reality, you can't reliably index without the risk of splitting surrogate pairs and the notion that UTF-16 is an "array of characters" is completely false at all levels. The memory representation may be "abstracted", but it's not even a leaky abstraction, it's a historical accident and a complete broken abstraction. Beware of believing what Microsoft say about C# string handling -- they're stuck with UTF-16, whether they like it or not.