2 ms·
Reminds me of Go strings: they usually store UTF-8 but they're actually 8-bit clean: "It's important to state right up front that a string holds arbitrary byte
by skybrian 11y ago
Reminds me of Go strings: they usually store UTF-8 but they're actually 8-bit clean:
"It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes."
https://blog.golang.org/strings https://blog.golang.org/strings
- geofft 11y agoIt's only 8-bit clean if you don't poke it very hard. Try either of the last two loop examples in that page after adding "\xff\x80" to the string; you get two (indistinguishable) U+FFFD REPLACEMENT CHARACTERs in the iteration. So the loop destroys data, which UTF-8B specifically does not. Also, it's a little disappointing that Go doesn't have a type-level way to say that a string is in fact UTF-8, not Latin-1 or something, and preferably that all values that inhabit that type are guaranteed to be valid and well-formed UTF-8. This is the cause of plenty of subtle bugs in Python 2, C, etc., which are all technically the result of programmer error, but in this decade, type systems should be helping us avoid common, subtle programmer errors.
- skybrian 11y agoIf you're juggling a bunch of different types of strings and want to keep them straight, Go does support defining separate types for them. An example is the HTML type from the template library [1]. The question is which sanitized string types are worth defining in the standard library. Presumably UTF-8 sanitized strings didn't make the cut. Not sure about UTF-8B. Suppose the input is already UTF-8B? Do you double-escape it somehow? It looks like DecodeRuneInString returns RuneError if it can't decode something and RuneError is defined as U+FFFD. The example uses a hard-coded string where it can't happen, so technically it's not a bug that it doesn't check for the error. But a linter might want to flag it. [1] http://golang.org/pkg/html/template/#HTML http://golang.org/pkg/html/template/#HTML