3 ms·

C++ programmer here: which one guarantees valid utf8, and why would a primitive container make guarantees about the values it's storing?

• lifthrasiir 4y ago
  Everything labelled as "string" is a valid UTF-8 string in Rust, and to my knowledge this decision was made very early in the history of Rust (before 0.1). Many "modern" languages (including modern enough C++) have a distinction between Unicode strings and byte strings however they are called and Rust just followed the suit.
  • M2Ys4U 4y ago
    OsStr and OsString aren't necessarily UTF-8, the data for those are "in the operating system’s preferred representation". str/String and CStr/CString are defined as UTF-8, though.
    • puffoflogic 4y ago
      > in the operating system’s preferred representation This is unfortunately misleading and a common misconception about OsStr. The documentation now explains: > OsStr losslessly represents a borrowed reference to a platform string. However, this representation is not necessarily in a form native to the platform What this means is that valid sequences of Unicode scalars are encoded as utf8 in OsStr on both Windows and Linux. The difference between OsStr and str is that the former can round-trip with the native encoding; that means that for Windows, there's a special way it encodes unpaired surrogates (wtf8) and on Linux it's actually just an arbitrary byte sequence. This choice of representation means that on every platform: you can always borrow an OsStr from a str (as_ref) and you can sometimes borrow a str from an OsStr (to_str). This cross-borrowing wouldn't be possible if OsStr were UCS-2 on Windows.
• Animats 4y ago
  "str" and "String" guarantee UTF-8. To make a String from an array of bytes, call pub fn from_utf8(vec: Vec<u8, Global>) -> Result<String, FromUtf8Error> which consumes the input Vec and returns it unmodified, if it's valid UTF-8,, or reports an error, if it's not. There are a number of related functions in this family. Such as pub fn from_utf8_lossy(v: &[u8]) -> Cow<'_, str> which takes in a slice of bytes and checks if it's a UTF-8 string. If it is, it returns the original str. Otherwise it makes a copy with any errors replaced with the Unicode error character. Vec<u8> and array slices such as &[u8] are primitive containers - they can store any sequence of u8 values. String is more like an object with access methods.
• pornel 4y ago
  The guarantee exists to speed up UTF-8 processing, so that it can safely assume working with whole codepoints/sequences (without extra out of bounds checks for every byte) and to ensure you can always losslessly roundtrip every string to and from other Unicode encodings without introducing any special notion of a broken character. There's also a security angle in this: text-processing algorithms may have different strategies for recovering from broken UTF-8, which could be exploited to fool parsers (e.g. if a 4-byte UTF-8 sequence has only 3 bytes matching, do you advance by 3 or 4 bytes?). Having the "valid UTF-8" state being part of the type system means it needs to be checked only once when the instance is created (which can be compile-time for constants), and doesn't have to be re-checked later, even if the string is mutated. Unlike a generic bag of bytes, the pubic interface on string won't allow making it invalid UTF-8.
• ntoskrnl 4y ago
  > why would a primitive container make guarantees about the values it's storing If you know you have valid UTF-8, you can safely skip bounds checks when decoding a codepoint that spans multiple bytes.
• afdbcreid 4y ago
  Note that if you don't want this guarantee, you can use [u8] and Vec<u8> (and Box<[u8]>). This does not support all string methods unfortunately, but there's work to do that.