5 ms·

Go's immutable UTF-8 string type is one of the nice things about the language A Go string is almost exactly like this C struct: struct String { uint8_

Go's immutable UTF-8 string type is one of the nice things about the language A Go string is almost exactly like this C struct: struct String { uint8_t* addr; ptrdiff_t len; }; The language guarantees you can't modify the bytes in memory range [addr, addr+len) Go's garbage collection makes it simple and natural to have one string alias ("point into", "overlap") part of another string. This works because strings are immutable. Compare this to the nightmare in C++, where substrings require copying or explicit handling The rune (UTF-8) iterator and other facilities make Unicode handling natural in Go In summary, Go's string type is a huge win

jrimbault 6y ago
I'd arguee Go's string type is "somewhat unusable"* since it doesn't enforce the guarantees it says/implies it does. The byte slice it points to is not guaranteed to be valid utf8. * of course to a degree, let's be reasonable, it's usable in a _lot_ of contexts, but I like my types to actually mean something.
- 37ef_ced3 6y ago
  In Go, malformed UTF-8 encodings are expected They are handled in a well-defined and graceful manner by all aspects of the language, runtime, and library
- DougBTX 6y ago
  Go doesn't guarantee any encoding for strings, very deliberately (so that, eg, they can be used to represent file names).
  - KMag 6y ago
    Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better representation for a file system path. That doesn't have great interaction with GUIs, but there are other corner cases that are often problematic for GUIs. In high school, one of my friends had a habit of putting games on the school library computers, and renaming them to names with non-printable characters using alt+number pad. (He used 129, IIRC, which isn't assigned a character in CP-1252.) The Windows 95 graphical shell would convert the non-printable characters to spaces for display, but when the librarian tried to delete the games, it would pass the display name to the kernel, which would complain that the presented path didn't exist.
    - jerf 6y ago
      It is not clear to me if you're elaborating or think you're disagreeing, but that is what Go does. It is generally assumed in Go that strings are UTF-8, but in practice what they actually are are just bags of bytes. Nothing really "UTF-y" will happen to them until you directly call UTF functions on them, which may produce new strings. It's something that I don't think could work unless your language is as recent as Go, and perhaps even Go 1.0 was pushing it, but it is an increasingly viable answer. For as thin as Go's encoding support really is in some sense, it has almost never caused me any trouble. The contexts where you are actively unsafe in assuming UTF-8 are decreasing, and the ones that are going to survive are the ones where there's some sort of explicit label, like in email. (Not that those are always trustworthy either.)
      - KMag 6y ago
        I'm saying it's useful to have valid strings and paths as separate types, but Go conflates the two types. Conflating the two is likely to lead to confused usage (such as programmers assuming there's a bijective mapping between valid paths and valid sequences of Unicode codepoints.) Pervasive confused usage of this sort in the wild in Python 2 was the motivation behind splitting bytes and strings in Python 3.
        jerf 6y ago
        As you pointed out, path types are awfully specialized to the OS and really even the file system itself. It is not clear that "Go" could provide such a thing. It doesn't need to, really, you can relatively easily create a type for the specific case you have. type PathSegment struct { path string // not exported, so only the empty one can be created externally } func MakePath(in string) (PathSegment, error) { // validate the input here } You'll need some more supporting types, of course, but it doesn't have to be provided by "Go" itself. (I have something rather like this in my codebase, though it is specialized to just Unix paths since I have no need to care about all the cross-platform details in this code base.) I wouldn't expect this to be something the language itself provides, and I'm not even that worried about it being missing from the standard library because it's awfully detail-oriented even for that.
- throwaway894345 6y ago
  I think of a string as an immutable byte slice. This is a little confusing since the language supports utf-8 literals only and it also lets you iterate over individual runes with for loops, but those are just conveniences over the fact that these are really just immutable byte slices. You could probably make your own “UTF8” type with the invariants you want (or at least someone would have to drop down into unsafe to violate the invariants) but in general Go programs don’t typically go that far, presumably because it doesn’t add much value in practice, which would suggest that your “somewhat unusable” claim (even with its caveat) is too strong. That said, I think it would be nice if Go made it a little easier/clearer to model a type that can only be created by a particular constructor or some such.
- skybrian 6y ago
  Go's standard library works with both possibly-malformed and verified UTF-8 strings, which is a nice property. The type system needed to explain what they actually do (take one of two possible input types and return the corresponding output type) would require generics, which we don't have yet. An alternative would be to duplicate the code to account for the different types, but we already have that for []byte versus string and that's bad enough already.
foo_barrio 6y ago
Java's sub-strings used to work sort of like this but was changed to use copy semantics. The structure was to have a "char[]" and an "offset" into that array. This allowed sub-strings to share the underlying array. However if you had a 1 char sub-string to a 1 GB array, the underlying array was never trimmed for garbage collection. In the case of a 1 char substring to a 1 GB string, is Go smart enough to free the rest of the array and keep only the 1 char?
- 37ef_ced3 6y ago
  In Go, as in C, there's no magic. A programmer using Go thinks of a string variable as a pointer/length pair, and knows what will happen. Just like with slices If you keep a pointer into an allocation (in your example, a small Go string pointing into a much larger Go string) the allocation is preserved by the garbage collector You should explicitly copy the substring out (instead of aliasing the underlying string) if retaining the underlying string causes you a problem
  - foo_barrio 6y ago
    Okay I understand. In Java, the default String class has the option to "intern" a string which just maintains a list of strings that can be shared. The change was made because often the devs were unaware of the string manipulation taking place in a third party library (eg XML/JSON/HTML parsing). You'd see the memory balloon, investigate and notice that String/char[] instances were dominating your heap. Instead of changing the entire implementation of the standard String class, they changed the semantics of the "substring()" call from O(1) to O(n) + memory side-effects.
- KMag 6y ago
  I wonder how hard the JVM folks looked into specialized weak references for solving this issue. The mark phase would treat all Strings with zero offset and full length as strongly referencing the byte[], and weakly otherwise. At the end of each full GC, you could iterate over all of your Strings (custom allocate/compact them to their own ranges of the heap for faster scanning), use some heuristics and probabilistic sampling to select some of the weakly reachable byte[]s for size reduction. A specialized copy/compact pass over the Strings could in-place replace byte[] references and fix up offsets. You'd probably also want to modify String.equals() to internally mutate equal strings to point to the same byte[], preferring smaller offsets, and when offsets are equal, preferring lower addresses. This is a light weight lazy version of the background String byte[] interning done by some JVMs.
Koshkin 6y ago
Oh well. The could've (should've?) used the layout of _bstr_t instead.

jrimbault 6y ago

I'd arguee Go's string type is "somewhat unusable"* since it doesn't enforce the guarantees it says/implies it does. The byte slice it points to is not guaranteed to be valid utf8. * of course to a degree, let's be reasonable, it's usable in a _lot_ of contexts, but I like my types to actually mean something.

37ef_ced3 6y ago
In Go, malformed UTF-8 encodings are expected They are handled in a well-defined and graceful manner by all aspects of the language, runtime, and library
DougBTX 6y ago
Go doesn't guarantee any encoding for strings, very deliberately (so that, eg, they can be used to represent file names).
- KMag 6y ago
  Filesystem paths are not strings. Linux doesn't enforce an encoding. Windows at least didn't used to enforce proper use of conjugate UTF-16 pairs (see WTF-8 encoding). I think OS X does perform UTF-8 normalization, which might include sanity checking and rejecting malformed UTF-8, but I'm not sure. A byte array (or a ref-counted singly-linked list of immutable byte arrays to save space/copying) is a much better representation for a file system path. That doesn't have great interaction with GUIs, but there are other corner cases that are often problematic for GUIs. In high school, one of my friends had a habit of putting games on the school library computers, and renaming them to names with non-printable characters using alt+number pad. (He used 129, IIRC, which isn't assigned a character in CP-1252.) The Windows 95 graphical shell would convert the non-printable characters to spaces for display, but when the librarian tried to delete the games, it would pass the display name to the kernel, which would complain that the presented path didn't exist.
  - jerf 6y ago
    It is not clear to me if you're elaborating or think you're disagreeing, but that is what Go does. It is generally assumed in Go that strings are UTF-8, but in practice what they actually are are just bags of bytes. Nothing really "UTF-y" will happen to them until you directly call UTF functions on them, which may produce new strings. It's something that I don't think could work unless your language is as recent as Go, and perhaps even Go 1.0 was pushing it, but it is an increasingly viable answer. For as thin as Go's encoding support really is in some sense, it has almost never caused me any trouble. The contexts where you are actively unsafe in assuming UTF-8 are decreasing, and the ones that are going to survive are the ones where there's some sort of explicit label, like in email. (Not that those are always trustworthy either.)
    - KMag 6y ago
      I'm saying it's useful to have valid strings and paths as separate types, but Go conflates the two types. Conflating the two is likely to lead to confused usage (such as programmers assuming there's a bijective mapping between valid paths and valid sequences of Unicode codepoints.) Pervasive confused usage of this sort in the wild in Python 2 was the motivation behind splitting bytes and strings in Python 3.
      - jerf 6y ago
        As you pointed out, path types are awfully specialized to the OS and really even the file system itself. It is not clear that "Go" could provide such a thing. It doesn't need to, really, you can relatively easily create a type for the specific case you have. type PathSegment struct { path string // not exported, so only the empty one can be created externally } func MakePath(in string) (PathSegment, error) { // validate the input here } You'll need some more supporting types, of course, but it doesn't have to be provided by "Go" itself. (I have something rather like this in my codebase, though it is specialized to just Unix paths since I have no need to care about all the cross-platform details in this code base.) I wouldn't expect this to be something the language itself provides, and I'm not even that worried about it being missing from the standard library because it's awfully detail-oriented even for that.
throwaway894345 6y ago
I think of a string as an immutable byte slice. This is a little confusing since the language supports utf-8 literals only and it also lets you iterate over individual runes with for loops, but those are just conveniences over the fact that these are really just immutable byte slices. You could probably make your own “UTF8” type with the invariants you want (or at least someone would have to drop down into unsafe to violate the invariants) but in general Go programs don’t typically go that far, presumably because it doesn’t add much value in practice, which would suggest that your “somewhat unusable” claim (even with its caveat) is too strong. That said, I think it would be nice if Go made it a little easier/clearer to model a type that can only be created by a particular constructor or some such.
skybrian 6y ago
Go's standard library works with both possibly-malformed and verified UTF-8 strings, which is a nice property. The type system needed to explain what they actually do (take one of two possible input types and return the corresponding output type) would require generics, which we don't have yet. An alternative would be to duplicate the code to account for the different types, but we already have that for []byte versus string and that's bad enough already.

foo_barrio 6y ago

Java's sub-strings used to work sort of like this but was changed to use copy semantics. The structure was to have a "char[]" and an "offset" into that array. This allowed sub-strings to share the underlying array. However if you had a 1 char sub-string to a 1 GB array, the underlying array was never trimmed for garbage collection. In the case of a 1 char substring to a 1 GB string, is Go smart enough to free the rest of the array and keep only the 1 char?

37ef_ced3 6y ago
In Go, as in C, there's no magic. A programmer using Go thinks of a string variable as a pointer/length pair, and knows what will happen. Just like with slices If you keep a pointer into an allocation (in your example, a small Go string pointing into a much larger Go string) the allocation is preserved by the garbage collector You should explicitly copy the substring out (instead of aliasing the underlying string) if retaining the underlying string causes you a problem
- foo_barrio 6y ago
  Okay I understand. In Java, the default String class has the option to "intern" a string which just maintains a list of strings that can be shared. The change was made because often the devs were unaware of the string manipulation taking place in a third party library (eg XML/JSON/HTML parsing). You'd see the memory balloon, investigate and notice that String/char[] instances were dominating your heap. Instead of changing the entire implementation of the standard String class, they changed the semantics of the "substring()" call from O(1) to O(n) + memory side-effects.
KMag 6y ago
I wonder how hard the JVM folks looked into specialized weak references for solving this issue. The mark phase would treat all Strings with zero offset and full length as strongly referencing the byte[], and weakly otherwise. At the end of each full GC, you could iterate over all of your Strings (custom allocate/compact them to their own ranges of the heap for faster scanning), use some heuristics and probabilistic sampling to select some of the weakly reachable byte[]s for size reduction. A specialized copy/compact pass over the Strings could in-place replace byte[] references and fix up offsets. You'd probably also want to modify String.equals() to internally mutate equal strings to point to the same byte[], preferring smaller offsets, and when offsets are equal, preferring lower addresses. This is a light weight lazy version of the background String byte[] interning done by some JVMs.

Koshkin 6y ago

Oh well. The could've (should've?) used the layout of _bstr_t instead.