3 ms·

> If you're happy with Go's "everything is a unicode string" approach then you should be happy to just treat everything as unicode. That's actually not really

> If you're happy with Go's "everything is a unicode string" approach then you should be happy to just treat everything as unicode. That's actually not really Go's approach. In Go, strings do not have encodings attached to them. Source files are defined to be UTF-8 (by the compiler), so string literals are always unicode. That's not quite the same thing as saying that the "string" type in Go is always Unicode (it's not). And when you're dealing with a byte slice ([]byte), you cannot make any assumptions about the encoding. It took a bit to wrap my head around this when I first read about it[0], but now that I think about it, I think it's the right way to go[1]. [0] http://blog.golang.org/strings http://blog.golang.org/strings [1] And for what it's worth, Go and UTF-8 were designed by (some of) the same people, so one would hope they'd get it right!

lmm 12y ago
You're right, I was lazily responding to the article on its own terms rather than engaging properly with go's string handling. I think the approach of keeping strings encoded has promise but it would need to be supported by a stronger type system than go's. When you're carrying around two encoded byte arrays it's really important that you know what their encodings are and don't try and e.g. concatenate them. Ruby can do this right because it can give you a runtime type error, but that's not acceptable in a compiled language. So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings. And you should really e.g. disallow slicing a byte array that represents a string, so that you don't cut a character in half. I know that one of the UTF-8 guys worked on go and I'm sure go will work well when everything's in UTF-8. But all languages work well when everything's in UTF-8; if anything this makes me more worried that go's authors won't give proper support to those who have to work with strings in non-UTF8 encodings. (By contrast one of the reasons Ruby's support is good is that the author is Japanese and therefore pretty much has to work with strings in multiple non-unicode encodings, because of han unification)
- burntsushi 12y ago
  > So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings. I don't buy this. You've just introduced some rather large complexity into the types of byte strings just for the sake of handling non-UTF8 cases, which seem to be getting less common than they used to be. Rust is taking a similar approach to Go (except that their string type must be valid UTF8). You can see a recent debate here: https://mail.mozilla.org/pipermail/rust-dev/2014-May/009725.html https://mail.mozilla.org/pipermail/rust-dev/2014-May/009725....
  - lmm 12y ago
    If you don't care about handling non-UTF8 cases, you can use pretty much any language (python3 included - the issues the OP is complaining about are when you have filenames in a different encoding from your terminal or the like), write the obvious thing, and it will work fine. For many use cases that's good enough. But the cases where languages are different, the cases where it gets interesting, are when that isn't enough. (And it won't be enough if you want to sell your software in Japan, for example).
    - pcwalton 12y ago
      There's actually a bit more to what Rust does: there is a well-known community library called rust-encoding that adds new string types that support various encodings. You can use this library if you need to support other encodings. The standard library supports only UTF-8, but it's simple enough to abstract over strings in multiple encodings if you need to (thanks to generics). I like this approach: it allows simplicity in the common case, for software that only needs to work in UTF-8, while allowing support for arbitrary other encodings. "Easy things should be easy, and hard things should be possible."
pcwalton 12y ago
> It took a bit to wrap my head around this when I first read about it[0], but now that I think about it, I think it's the right way to go[1]. I don't actually think it's the right way to go: I'd prefer if strings enforced that they were UTF-8. That saves a lot of error checking by clients that consume it and want to be defensive. It's most efficient to have the type system do the work.
- andreasvc 12y ago
  So how do you then deal with problem in the original blog post: a filename that is a non-UTF8 byte string? Surrogates? A separate byte string type?
  - pcwalton 12y ago
    Use a separate byte string type. The problem is that paths are not UTF-8. That's a fixable problem: just let the type system know about it.

lmm 12y ago

You're right, I was lazily responding to the article on its own terms rather than engaging properly with go's string handling. I think the approach of keeping strings encoded has promise but it would need to be supported by a stronger type system than go's. When you're carrying around two encoded byte arrays it's really important that you know what their encodings are and don't try and e.g. concatenate them. Ruby can do this right because it can give you a runtime type error, but that's not acceptable in a compiled language. So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings. And you should really e.g. disallow slicing a byte array that represents a string, so that you don't cut a character in half. I know that one of the UTF-8 guys worked on go and I'm sure go will work well when everything's in UTF-8. But all languages work well when everything's in UTF-8; if anything this makes me more worried that go's authors won't give proper support to those who have to work with strings in non-UTF8 encodings. (By contrast one of the reasons Ruby's support is good is that the author is Japanese and therefore pretty much has to work with strings in multiple non-unicode encodings, because of han unification)

burntsushi 12y ago
> So you need to distinguish between byte arrays with statically known encodings, byte arrays with dynamically known encodings and byte arrays with unknown encodings. I don't buy this. You've just introduced some rather large complexity into the types of byte strings just for the sake of handling non-UTF8 cases, which seem to be getting less common than they used to be. Rust is taking a similar approach to Go (except that their string type must be valid UTF8). You can see a recent debate here: https://mail.mozilla.org/pipermail/rust-dev/2014-May/009725.html https://mail.mozilla.org/pipermail/rust-dev/2014-May/009725....
- lmm 12y ago
  If you don't care about handling non-UTF8 cases, you can use pretty much any language (python3 included - the issues the OP is complaining about are when you have filenames in a different encoding from your terminal or the like), write the obvious thing, and it will work fine. For many use cases that's good enough. But the cases where languages are different, the cases where it gets interesting, are when that isn't enough. (And it won't be enough if you want to sell your software in Japan, for example).
  - pcwalton 12y ago
    There's actually a bit more to what Rust does: there is a well-known community library called rust-encoding that adds new string types that support various encodings. You can use this library if you need to support other encodings. The standard library supports only UTF-8, but it's simple enough to abstract over strings in multiple encodings if you need to (thanks to generics). I like this approach: it allows simplicity in the common case, for software that only needs to work in UTF-8, while allowing support for arbitrary other encodings. "Easy things should be easy, and hard things should be possible."

pcwalton 12y ago

> It took a bit to wrap my head around this when I first read about it[0], but now that I think about it, I think it's the right way to go[1]. I don't actually think it's the right way to go: I'd prefer if strings enforced that they were UTF-8. That saves a lot of error checking by clients that consume it and want to be defensive. It's most efficient to have the type system do the work.

andreasvc 12y ago
So how do you then deal with problem in the original blog post: a filename that is a non-UTF8 byte string? Surrogates? A separate byte string type?
- pcwalton 12y ago
  Use a separate byte string type. The problem is that paths are not UTF-8. That's a fixable problem: just let the type system know about it.