9 ms·

Go was designed by some old-school folks that maybe stuck a bit too hard to their principles, losing sight of the practical conveniences. I'd say that it's ent

• nasretdinov 1y ago
  Note that Go strings can be invalid UTF-8, they dropped panicking on encountering an invalid UTF string before 1.0 I think
  • xyzzyz 1y ago
    This also epitomizes the issue. What's the point of having `string` type at all, if it doesn't allow you to make any extra assumptions about the contents beyond `[]byte`? The answer is that they planned to make conversion to `string` error out when it's invalid UTF-8, and then assume that `string`s are valid UTF-8, but then it caused problems elsewhere, so they dropped it for immediate practical convenience.
    • assbuttbuttass 1y ago
      string is just an immutable []byte. It's actually one of my favorite things about Go that strings can contain invalid utf-8, so you don't end up with the Rust mess of String vs OSString vs PathBuf vs Vec<u8>. It's all just string
      • zozbot234 1y ago
        Rust &str and String are specifically intended for UTF-8 valid text. If you're working with arbitrary byte sequences, that's what &[u8] and Vec<u8> are for in Rust. It's not a "mess", it's just different from what Golang does.
        • gf000 1y ago
          If anything that will make Rust programs likely to be correct under any strange text input, while Go might just handle the happy path of ASCII inputs. Stuff like this matters a great deal on the standard library level.
          • deleted 1y ago
            [deleted]
        • maxdamantus 1y ago
          It's never been clear to me where such a type is actually useful. In what cases do you really need to restrict it to valid UTF-8? You should always be able to iterate the code points of a string, whether or not it's valid Unicode. The iterator can either silently replace any errors with replacement characters, or denote the errors by returning eg, `Result<char, Utf8Error>`, depending on the use case. All languages that have tried restricting Unicode afaik have ended up adding workarounds for the fact that real world "text" sometimes has encoding errors and it's often better to just preserve the errors instead of corrupting the data through replacement characters, or just refusing to accept some inputs and crashing the program. In Rust there's bstr/ByteStr (currently being added to std), awkward having to decide which string type to use. In Python there's PEP-383/"surrogateescape", which works because Python strings are not guaranteed valid (they're potentially ill-formed UTF-32 sequences, with a range restriction). Awkward figuring out when to actually use it. In Raku there's UTF8-C8, which is probably the weirdest workaround of all (left as an exercise for the reader to try to understand .. oh, and it also interferes with valid Unicode that's not normalized, because that's another stupid restriction). Meanwhile the Unicode standard itself specifies Unicode strings as being sequences of code units [0][1], so Go is one of the few modern languages that actually implements Unicode (8-bit) strings. Note that at least two out of the three inventors of Go also basically invented UTF-8. [0] https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G32765 https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha... > Unicode string: A code unit sequence containing code units of a particular Unicode encoding form. [1] https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G32860 https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha... > Unicode strings need not contain well-formed code unit sequences under all conditions. This is equivalent to saying that a particular Unicode string need not be in a Unicode encoding form.
    • roncesvalles 1y ago
      I've always thought the point of the string type was for indexing. One index of a string is always one character, but characters are sometimes composed of multiple bytes.
      • birn559 1y ago
        You can't do that in a performant way and going that route can lead to problems, because characters (= graphemes in the language of Unicode) generally don't always behave as developers assume.
      • crazygringo 1y ago
        Yup. But to be clear, in Unicode a string will index code points, not characters. E.g. a single emoji can be made of multiple code points, as well as certain characters in certain languages. The Unicode name for a character like this is a "grapheme", and grapheme splitting is so complicated it generally belongs in a dedicated Unicode library, not a general-purpose string object.
    • 0x000xca0xfe 1y ago
      Why not use utf8.ValidString in the places it is needed? Why burden one of the most basic data types with highly specific format checks? It's far better to get some � when working with messy data instead of applications refusing to work and erroring out left and right.
      • const_cast 1y ago
        IMO utf8 isn't a highly specific format, it's universal for text. Every ascii string you'd write in C or C++ or whatever is already utf8. So that means that for 99% of scenarios, the difference between char[] and a proper utf8 string is none. They have the same data representation and memory layout. The problem comes in when people start using string like they use string in PHP. They just use it to store random bytes or other binary data. This makes no sense with the string type. String is text, but now we don't have text. That's a problem. We should use byte[] or something for this instead of string. That's an abuse of string. I don't think allowing strings to not be text is too constraining - that's what a string is!
        • adastra22 1y ago
          Not all text is UTF-8, and there are real world contexts (e.g. Windows) where this matters a lot.
          • const_cast 1y ago
            Yes, Windows text is broken in its own special way. We can try to shove it into objects that work on other text but this won't work in edge cases. Like if I take text on Linux and try to write a Windows file with that text, it's broken. And vice versa. Go let's you do the broken thing. In Rust or even using libraries in most languages, you cant. You have to specifically convert between them. That's why I mean when I say "storing random binary data as text". Sure, Windows almost UTF16 abomination is kind of text, but not really. Its its own thing. That requires a different type of string OR converting it to a normal string.
            • 1y ago
    • naikrovek 1y ago
      I think maybe you've forgotten about the rune type. Rune does make assumptions. []Rune is for sequences of UTF characters. rune is an alias for int32. string, I think, is an alias for []byte.
      • TheDong 1y ago
        `string` is not an alias for []byte. Consider: for i, chr := range string([]byte{226, 150, 136, 226, 150, 136}) { fmt.Printf("%d = %v\n", i, chr) // note, s[i] != chr } How many times does that loop over 6 bytes iterate? The answer is it iterates twice, with i=0 and i=3. There's also quite a few standard APIs that behave weirdly if a string is not valid utf-8, which wouldn't be the case if it was just a bag of bytes.
        • naikrovek 1y ago
          Go programmers (and `range`) assume that string is always valid UTF-8 but there is no guarantee by the language that a string is valid UTF-8. The string itself is still a []byte. `range` sees the `string` type and has special handling for strings that it does not have when it ranges over []byte. Recall that aliased types are not viewed as the same type at any time. A couple quotes from the Go Blog by Rob Pike: > It’s important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes. > Besides the axiomatic detail that Go source code is UTF-8, there’s really only one way that Go treats UTF-8 specially, and that is when using a for range loop on a string. Both from https://go.dev/blog/strings https://go.dev/blog/strings If you want UTF-8 in a guaranteed way, use the functions available in unicode/utf8 for that. Using `string` is not sufficient unless you make sure you only put UTF-8 into those strings. If you put valid UTF-8 into a string, you can be sure that the string holds valid UTF-8, but if someone else puts data into a string, and you assume that it is valid UTF-8, you may have a problem because of that assumption.
    • tialaramex 1y ago
      Rust apparently got relatively close to not having &str as a primitive type and instead only providing a library alias to &[u8] when Rust 1.0 shipped. Score another for Rust's Safety Culture. It would be convenient to just have &str as an alias for &[u8] but if that mistake had been allowed all the safety checking that Rust now does centrally has to be owned by every single user forever. Instead of a few dozen checks overseen by experts there'd be myriad sprinkled across every project and always ready to bite you.
      • adastra22 1y ago
        . (early morning brain fart -- I need my coffee)
        • tialaramex 1y ago
          So it's true that technically the primitive type is str, and indeed it's even possible to make a &mut str though it's quite rare that you'd want to mutably borrow the string slice. However no &str is not "an alias for &&String" and I can't quite imagine how you'd think that. String doesn't exist in Rust's core, it's from alloc and thus wouldn't be available if you don't have an allocator.
          • zozbot234 1y ago
            str is not really a "primitive type", it only exists abstractly as an argument to type constructors - treating the & operator as a "type constructor" for that purpose, but including Box<>, Rc<>, Arc<> etc. So you can have Box<str> or Arc<str> in addition to &str or perhaps &mut str, but not really 'str' in isolation.
      • inferiorhuman 1y ago
        Even so you end up with paper cuts like len which returns the number of bytes.
        • toast0 1y ago
          The problem with string length is there's probably at least four concepts that could conceivably be called length, and few people are happy when none of them are len. Of the top of my head, in order of likely difficulty to calculate: byte length, number of code points, number of grapheme/characters, height/width to display. Maybe it would be best for Str not to have len at all. It could have bytes, code_points, graphemes. And every use would be precise.
    • Ferret7446 1y ago
      > if it doesn't allow you to make any extra assumptions about the contents beyond `[]byte`? It does though? Strings are internable, comparable, can be keys, etc.
• koakuma-chan 1y ago
  > What if the file name is not valid UTF-8 Nothing? Neither Go nor the OS require file names to be UTF-8, I believe
  • deleted 1y ago
    [deleted]
  • johncolanduoni 1y ago
    Well, Windows is an odd beast when 8-bit file names are used. If done naively, you can’t express all valid filenames with even broken UTF-8 and non-valid-Unicode filenames cannot be encoded to UTF-8 without loss or some weird convention. You can do something like WTF-8 (not a misspelling, alas) to make it bidirectional. Rust does this under the hood but doesn’t expose the internal representation.
    • andyferris 1y ago
      I believe the same is true on linux, which only cares about 0x2f bytes (i.e. /)
      • matt_kantor 1y ago
        And 0x00.
      • orthoxerox 1y ago
        And 0x00, if I remember correctly.
      • johncolanduoni 1y ago
        Windows paths are not necessarily well-formed UTF-16 (UCS-2 by some people’s definition) down to the filesystem level. If they were always well formed, you could convert to a single byte representation by straightforward Unicode re-encoding. But since they aren’t - there are choices that need to be made about what to do with malformed UTF-16 if you want to round trip them to single byte strings such that they match UTF-8 encoding if they are well formed. In Linux, they’re 8-bit almost-arbitrary strings like you noted, and usually UTF-8. So they always have a convenient 8-bit encoding (I.e. leave them alone). If you hated yourself and wanted to convert them to UTF-16, however, you’d have the same problem Windows does but in reverse.
• ants_everywhere 1y ago
  > they stuck to the practical convenience of solving the problem that they had in front of them, quickly, instead of analyzing the problem from the first principles, and solving the problem correctly (or using a solution that was Not Invented Here). I've said this before, but much of Go's design looks like it's imitating the C++ style at Google. The comments where I see people saying they like something about Go it's often an idiom that showed up first in the C++ macros or tooling. I used to check this before I left Google, and I'm sure it's becoming less true over time. But to me it looks like the idea of Go was basically "what if we created a Python-like compiled language that was easier to onboard than C++ but which still had our C++ ergonomics?"
  • shrubble 1y ago
    Didn’t Go come out of a language that was written for Plan9, thus pre-dating Rob Pike’s work at Google?
    • ants_everywhere 1y ago
      not that I recall but I may not be recalling correctly. But certainly, anyone will bring their previous experience to the project, so there must be some Plan9 influence in there somewhere
      • kragen 1y ago
        They were literally using the Plan9 C compiler and linker.
        • ants_everywhere 1y ago
          Yes I'm aware
          • kragen 1y ago
            Literally building the project out of the Plan 9 source code is very far from "bring[ing] their previous experience to the project, (...) some Plan9 influence in there somewhere"
• jerf 1y ago
  While the general question about string encoding is fine, unfortunately in a general-purpose and cross-platform language, a file interface that enforces Unicode correctness is actively broken, in that there are files out in the world it will be unable to interact with. If your language is enforcing that, and it doesn't have a fallback to a bag of bytes, it is broken, you just haven't encountered it. Go is correct on this specific API. I'm not celebrating that fact here, nor do I expect the Go designers are either, but it's still correct.
  • klodolph 1y ago
    This is one of those things that kind of bugs me about, say, OsStr / OsString in Rust. In theory, it’s a very nice, principled approach to strings (must be UTF-8) and filenames (arbitrary bytes, almost, on Linux & Mac). In practice, the ergonomics around OsStr are horrible. They are missing most of the API that normal strings have… it seems like manipulating them is an afterthought, and it was assumed that people would treat them as opaque (which is wrong). Go’s more chaotic approach to allow strings to have non-Unicode contents is IMO more ergonomic. You validate that strings are UTF-8 at the place where you care that they are UTF-8. (So I’m agreeing.)
    • Kinrany 1y ago
      > You validate that strings are UTF-8 at the place where you care that they are UTF-8. The problem with this, as with any lack of static typing, is that you now have to validate at _every_ place that cares, or to carefully track whether a value has already been validated, instead of validating once and letting the compiler check that it happened.
      • klodolph 1y ago
        In practice, the validation generally happens when you convert to JSON or use an HTML template or something like that, so it’s not so many places. Validation is nice but Rust’s principled approach leaves me high and dry sometimes. Maybe Rust will finish figuring out the OsString interface and at that point we can say Rust has “won” the conversation, but it’s not there yet, and it’s been years.
• herbstein 1y ago
  Much more egregious is the fact that the API allows returning both an error and a valid file handle. That may be documented to not happen. But look at the Read method instead. It will return both errors and a length you need to handle at the same time.
  • nasretdinov 1y ago
    The Read() method is certainly an exception rather than a rule. The common convention is to return nil value upon encountering an error unless there's real value in returning both, e.g. for a partial read that failed in the end but produced some non-empty result nevertheless. It's a rare occasion, yes, but if you absolutely have to handle this case you can. Otherwise you typically ignore the result if err!=nil. It's a mess, true, but real world is also quite messy unfortunately, and Go acknowledges that
    • stouset 1y ago
      Go doesn't acknowledge that. It punts. Most of the time if there's a result, there's no error. If there's an error, there's no result. But don't forget to check every time! And make sure you don't make a mistake when you're checking and accidentally use the value anyway, because even though it's technically meaningless it's still nominally a meaningful value since zero values are supposed to be meaningful. Oh and make sure to double-check the docs, because the language can't let you know about the cases where both returns are meaningful. The real world is messy. And golang doesn't give you advance warning on where the messes are, makes no effort to prevent you from stumbling into them, and stands next to you constantly criticizing you while you clean them up by yourself. "You aren't using that variable any more, clean that up too." "There's no new variables now, so use `err =` instead of `err :=`."
• silverwind 1y ago
  > What if the file name is not valid UTF-8, though They could support passing filename as `string | []byte`. But wait, go does not even have union types.
  • lblume 1y ago
    But []byte, or a wrapper like Path, is enough, if strings are easily convertible into it. Rust does it that way via the AsRef<T> trait.
• deleted 1y ago
  [deleted]
• stouset 1y ago
  [flagged]
  • blibble 1y ago
    > Golang makes it easy to do the dumb, wrong, incorrect thing that looks like it works 99.7% of the time. How can that be wrong? It works in almost all cases! my favorite example of this was the go authors refusing to add monotonic time into the standard library because they confidently misunderstood its necessity (presumably because clocks at google don't ever step) then after some huge outages (due to leap seconds) they finally added it now the libraries are a complete a mess because the original clock/time abstractions weren't built with the concept of multiple clocks and every go program written is littered with terrible bugs due to use of the wrong clock https://github.com/golang/go/issues/12914 https://github.com/golang/go/issues/12914 (https://github.com/golang/go/issues/12914#issuecomment-150754067 https://github.com/golang/go/issues/12914#issuecomment-15075... might qualify for the worst comment ever)
    • 0cf8612b2e1e 1y ago
      This issue is probably my favorite Goism. Real issue identified and the feedback is, “You shouldn’t run hardware that way. Run servers like Google does without time jumping.” Similar with the original stance to code versioning. Just run a monorepo!
  • jen20 1y ago
    I can count on fewer hands the number of times I've been bitten by such things in over 10 years of professional Go vs bitten just in the last three weeks by half-assed Java.
    • deleted 1y ago
      [deleted]
    • gf000 1y ago
      There is a lot to say about Java, but the libraries (both standard lib and popular third-party ones) are goddamn battle-hardened, so I have a hard time believing your claim.
• kragen 1y ago
  If the filename is not valid UTF-8, Golang can still open the file without a problem, as long as your filesystem doesn't attempt to be clever. Linux ext4fs and Go both consider filenames to be binary strings except that they cannot contain NULs. This is one of the minor errors in the post.
• perryizgr8 1y ago
  > What if the file name is not valid UTF-8, though? Then make it valid UTF-8. If you try to solve the long tail of issues in a commonly used function of the library its going to cause a lot of pain. This approach is better. If someone has a weird problem like file names with invalid characters, they can solve it themselves, even publish a package. Why complicate 100% of uses for solving 0.01% of issues?
  • nomel 1y ago
    > Then make it valid UTF-8. I think you misunderstand. How do you do that for a file that exists on disk that's trying to be read? Rename it for them? They may not like that.