6 ms·

This also epitomizes the issue. What's the point of having `string` type at all, if it doesn't allow you to make any extra assumptions about the contents beyond

• assbuttbuttass 1y ago
  string is just an immutable []byte. It's actually one of my favorite things about Go that strings can contain invalid utf-8, so you don't end up with the Rust mess of String vs OSString vs PathBuf vs Vec<u8>. It's all just string
  • zozbot234 1y ago
    Rust &str and String are specifically intended for UTF-8 valid text. If you're working with arbitrary byte sequences, that's what &[u8] and Vec<u8> are for in Rust. It's not a "mess", it's just different from what Golang does.
    • gf000 1y ago
      If anything that will make Rust programs likely to be correct under any strange text input, while Go might just handle the happy path of ASCII inputs. Stuff like this matters a great deal on the standard library level.
      • deleted 1y ago
        [deleted]
    • maxdamantus 1y ago
      It's never been clear to me where such a type is actually useful. In what cases do you really need to restrict it to valid UTF-8? You should always be able to iterate the code points of a string, whether or not it's valid Unicode. The iterator can either silently replace any errors with replacement characters, or denote the errors by returning eg, `Result<char, Utf8Error>`, depending on the use case. All languages that have tried restricting Unicode afaik have ended up adding workarounds for the fact that real world "text" sometimes has encoding errors and it's often better to just preserve the errors instead of corrupting the data through replacement characters, or just refusing to accept some inputs and crashing the program. In Rust there's bstr/ByteStr (currently being added to std), awkward having to decide which string type to use. In Python there's PEP-383/"surrogateescape", which works because Python strings are not guaranteed valid (they're potentially ill-formed UTF-32 sequences, with a range restriction). Awkward figuring out when to actually use it. In Raku there's UTF8-C8, which is probably the weirdest workaround of all (left as an exercise for the reader to try to understand .. oh, and it also interferes with valid Unicode that's not normalized, because that's another stupid restriction). Meanwhile the Unicode standard itself specifies Unicode strings as being sequences of code units [0][1], so Go is one of the few modern languages that actually implements Unicode (8-bit) strings. Note that at least two out of the three inventors of Go also basically invented UTF-8. [0] https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G32765 https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha... > Unicode string: A code unit sequence containing code units of a particular Unicode encoding form. [1] https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G32860 https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha... > Unicode strings need not contain well-formed code unit sequences under all conditions. This is equivalent to saying that a particular Unicode string need not be in a Unicode encoding form.
      • xyzzyz 1y ago
        The way Rust handles this is perfectly fine. String type promises its contents are valid UTF-8. When you create it from array of bytes, you have three options: 1) ::from_utf8, which will force you to handle invalid UTF-8 error, 2) ::from_utf8_lossy, which will replace invalid code points with replacement character code point, and 3) from_utf8_unchecked, which will not do the validity check and is explicitly marked as unsafe.
        • maxdamantus 1y ago
          But there's no option to just construct the string with the invalid bytes. 3) is not for this purpose; it is for when you already know that it is valid. If you use 3) to create a &str/String from invalid bytes, you can't safely use that string as the standard library is unfortunately designed around the assumption that only valid UTF-8 is stored. https://doc.rust-lang.org/std/primitive.str.html#invariant https://doc.rust-lang.org/std/primitive.str.html#invariant > Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.
          • adastra22 1y ago
            I don’t understand this complaint. (3) sounds like exactly what you are asking for. And yes, doing unsafe thing is unsafe.
            • maxdamantus 1y ago
              > I don’t understand this complaint. (3) sounds like exactly what you are asking for. And yes, doing unsafe thing is unsafe You're meant to use `unsafe` as a way of limiting the scope of reasoning about safety. Once you construct a `&str` using `from_utf8_unchecked`, you can't safely pass it to any other function without looking at its code and reasoning about whether it's still safe. Also see the actual documentation: https://doc.rust-lang.org/std/primitive.str.html#method.from_utf8_unchecked https://doc.rust-lang.org/std/primitive.str.html#method.from... > Safety: The bytes passed in must be valid UTF-8.
• roncesvalles 1y ago
  I've always thought the point of the string type was for indexing. One index of a string is always one character, but characters are sometimes composed of multiple bytes.
  • birn559 1y ago
    You can't do that in a performant way and going that route can lead to problems, because characters (= graphemes in the language of Unicode) generally don't always behave as developers assume.
  • crazygringo 1y ago
    Yup. But to be clear, in Unicode a string will index code points, not characters. E.g. a single emoji can be made of multiple code points, as well as certain characters in certain languages. The Unicode name for a character like this is a "grapheme", and grapheme splitting is so complicated it generally belongs in a dedicated Unicode library, not a general-purpose string object.
• 0x000xca0xfe 1y ago
  Why not use utf8.ValidString in the places it is needed? Why burden one of the most basic data types with highly specific format checks? It's far better to get some � when working with messy data instead of applications refusing to work and erroring out left and right.
  • const_cast 1y ago
    IMO utf8 isn't a highly specific format, it's universal for text. Every ascii string you'd write in C or C++ or whatever is already utf8. So that means that for 99% of scenarios, the difference between char[] and a proper utf8 string is none. They have the same data representation and memory layout. The problem comes in when people start using string like they use string in PHP. They just use it to store random bytes or other binary data. This makes no sense with the string type. String is text, but now we don't have text. That's a problem. We should use byte[] or something for this instead of string. That's an abuse of string. I don't think allowing strings to not be text is too constraining - that's what a string is!
    • adastra22 1y ago
      Not all text is UTF-8, and there are real world contexts (e.g. Windows) where this matters a lot.
      • const_cast 1y ago
        Yes, Windows text is broken in its own special way. We can try to shove it into objects that work on other text but this won't work in edge cases. Like if I take text on Linux and try to write a Windows file with that text, it's broken. And vice versa. Go let's you do the broken thing. In Rust or even using libraries in most languages, you cant. You have to specifically convert between them. That's why I mean when I say "storing random binary data as text". Sure, Windows almost UTF16 abomination is kind of text, but not really. Its its own thing. That requires a different type of string OR converting it to a normal string.
        • 1y ago
• naikrovek 1y ago
  I think maybe you've forgotten about the rune type. Rune does make assumptions. []Rune is for sequences of UTF characters. rune is an alias for int32. string, I think, is an alias for []byte.
  • TheDong 1y ago
    `string` is not an alias for []byte. Consider: for i, chr := range string([]byte{226, 150, 136, 226, 150, 136}) { fmt.Printf("%d = %v\n", i, chr) // note, s[i] != chr } How many times does that loop over 6 bytes iterate? The answer is it iterates twice, with i=0 and i=3. There's also quite a few standard APIs that behave weirdly if a string is not valid utf-8, which wouldn't be the case if it was just a bag of bytes.
    • naikrovek 1y ago
      Go programmers (and `range`) assume that string is always valid UTF-8 but there is no guarantee by the language that a string is valid UTF-8. The string itself is still a []byte. `range` sees the `string` type and has special handling for strings that it does not have when it ranges over []byte. Recall that aliased types are not viewed as the same type at any time. A couple quotes from the Go Blog by Rob Pike: > It’s important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes. > Besides the axiomatic detail that Go source code is UTF-8, there’s really only one way that Go treats UTF-8 specially, and that is when using a for range loop on a string. Both from https://go.dev/blog/strings https://go.dev/blog/strings If you want UTF-8 in a guaranteed way, use the functions available in unicode/utf8 for that. Using `string` is not sufficient unless you make sure you only put UTF-8 into those strings. If you put valid UTF-8 into a string, you can be sure that the string holds valid UTF-8, but if someone else puts data into a string, and you assume that it is valid UTF-8, you may have a problem because of that assumption.
• tialaramex 1y ago
  Rust apparently got relatively close to not having &str as a primitive type and instead only providing a library alias to &[u8] when Rust 1.0 shipped. Score another for Rust's Safety Culture. It would be convenient to just have &str as an alias for &[u8] but if that mistake had been allowed all the safety checking that Rust now does centrally has to be owned by every single user forever. Instead of a few dozen checks overseen by experts there'd be myriad sprinkled across every project and always ready to bite you.
  • adastra22 1y ago
    . (early morning brain fart -- I need my coffee)
    • tialaramex 1y ago
      So it's true that technically the primitive type is str, and indeed it's even possible to make a &mut str though it's quite rare that you'd want to mutably borrow the string slice. However no &str is not "an alias for &&String" and I can't quite imagine how you'd think that. String doesn't exist in Rust's core, it's from alloc and thus wouldn't be available if you don't have an allocator.
      • zozbot234 1y ago
        str is not really a "primitive type", it only exists abstractly as an argument to type constructors - treating the & operator as a "type constructor" for that purpose, but including Box<>, Rc<>, Arc<> etc. So you can have Box<str> or Arc<str> in addition to &str or perhaps &mut str, but not really 'str' in isolation.
  • inferiorhuman 1y ago
    Even so you end up with paper cuts like len which returns the number of bytes.
    • toast0 1y ago
      The problem with string length is there's probably at least four concepts that could conceivably be called length, and few people are happy when none of them are len. Of the top of my head, in order of likely difficulty to calculate: byte length, number of code points, number of grapheme/characters, height/width to display. Maybe it would be best for Str not to have len at all. It could have bytes, code_points, graphemes. And every use would be precise.
• Ferret7446 1y ago
  > if it doesn't allow you to make any extra assumptions about the contents beyond `[]byte`? It does though? Strings are internable, comparable, can be keys, etc.