4 ms·

In Go, strings are UTF8, but when you iterate over UTF8 characters like this: for _, c := range str { } variable "c" is 32-bit. So it's not really wastefu

by kgeist 3y ago

In Go, strings are UTF8, but when you iterate over UTF8 characters like this: for _, c := range str { } variable "c" is 32-bit. So it's not really wasteful, it fits in a register. It can be wasteful if you declare a slice of characters, like []rune, but I don't remember ever seeing it in practice.

aardvark179 3y ago
In Java strings are stored in the heap either as compact strings for cases where everything fits in 8 bits, or as UTF-16. Iterating over them using chars will give you those 16 bit things, but there are other methods you can use to iterate over them as code points, just like you can in Go. The old char behaviour may not be what you really want, but it can’t be changed without breaking existing code.
- LukeShu 3y ago
  Right. But this thread is about "OK, but what if we forked Java and could make breaking changes like that." Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.
erik_seaberg 3y ago
Most of the things you could do with a single codepoint aren’t valid unless you search for a group of unicode.IsMark(c) and process them together with the previous codepoint. Now I’m concerned that I don’t see that in the standard library. https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...
- LukeShu 3y ago
  Yes, but also no. You are correct that there are lots of things that you can't do without grouping codepoints into grapheme clusters. And it is unfortunate that the Go standard library doesn't have utilities for grouping codepoints into grapheme clusters; the Go team's response to that is largely "don't do those things; even if we had utilities for grouping codepoints into grapheme clusters, those things would be hard to do and you are likely to get it wrong; human-level text is complicated, so treat it as opaque as much as you can." But also, there are still lots of things you can do with just codepoints. You can tokenize. You can trim whitespace. You can do case-conversions and comparisons. You can transcode to a different encoding (whether that be a different Unicode encoding, or something like a JSON where some codepoints will need to be \uXXXX-encoded). So yeah, codepoints aren't always what you want, but they're what you want more often than Java's "usually codepoints, but sometimes surrogate pairs". Especially because I suspect that very few Java programmers bother to test their code with surrogate pairs.
  - erik_seaberg 3y ago
    I agree that iterating over UTF-16 code units can never be less painful than iterating over UTF-32 codepoints, but to be fair Java (like Win32) comes from more than fourteen years earlier when we still hoped UCS-2 would be enough for the world.

by kgeist 3y ago

aardvark179 3y ago

In Java strings are stored in the heap either as compact strings for cases where everything fits in 8 bits, or as UTF-16. Iterating over them using chars will give you those 16 bit things, but there are other methods you can use to iterate over them as code points, just like you can in Go. The old char behaviour may not be what you really want, but it can’t be changed without breaking existing code.

LukeShu 3y ago
Right. But this thread is about "OK, but what if we forked Java and could make breaking changes like that." Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.

erik_seaberg 3y ago

Most of the things you could do with a single codepoint aren’t valid unless you search for a group of unicode.IsMark(c) and process them together with the previous codepoint. Now I’m concerned that I don’t see that in the standard library. https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...

LukeShu 3y ago
Yes, but also no. You are correct that there are lots of things that you can't do without grouping codepoints into grapheme clusters. And it is unfortunate that the Go standard library doesn't have utilities for grouping codepoints into grapheme clusters; the Go team's response to that is largely "don't do those things; even if we had utilities for grouping codepoints into grapheme clusters, those things would be hard to do and you are likely to get it wrong; human-level text is complicated, so treat it as opaque as much as you can." But also, there are still lots of things you can do with just codepoints. You can tokenize. You can trim whitespace. You can do case-conversions and comparisons. You can transcode to a different encoding (whether that be a different Unicode encoding, or something like a JSON where some codepoints will need to be \uXXXX-encoded). So yeah, codepoints aren't always what you want, but they're what you want more often than Java's "usually codepoints, but sometimes surrogate pairs". Especially because I suspect that very few Java programmers bother to test their code with surrogate pairs.
- erik_seaberg 3y ago
  I agree that iterating over UTF-16 code units can never be less painful than iterating over UTF-32 codepoints, but to be fair Java (like Win32) comes from more than fourteen years earlier when we still hoped UCS-2 would be enough for the world.