3 ms·

I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, of

I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, often commits the sin of requiring unnecessary processing. Your CSV example here is a pretty good one: you either need heuristics or some kind of OOB data to "know" if you've got an encoding that can possibly represent a CSV file. If you've got that, you can always at least convert to a compatible encoding, e.g. UTF-8. If you don't, you can either bail or cross your fingers and hope that it'll work out (this is perfectly reasonable in many cases). Regardless, if you want to give programmers all of these options with zero cost you need a fully functional encoding-agnostic byte string type. I do think you've done the legwork to improve the situation though? It would be cool if Rust incorporated bstr into std; I think that would give library authors (and maybe std authors) confidence in working on a lower level interface that didn't guarantee/require valid UTF-8. > Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-) 100% agree. Honestly, re-reading OP it's clear I got really hung up on the bullet points in the middle ("Do you know how to render it" etc.), but it is really about UTF-8 in programming language string types. And hey, empirically it worked haha :)

burntsushi 2y ago
Yeah I think there is some desire for bstr to come into std. But... it's like... tricky. We're steadily adding "string-like" methods to `&[u8]`, but for example, we still lack substring search on `&[u8]`. And lots of other things too. And there are things to consider like the fact that std has this elaborate `Pattern` abstraction (which is itself unstable, but still permits ergonomic polymorphic use of routines that are generic over it). Does that also need to get moved to `&[u8]`? And then there are things like `FromStr` that std implements for lots of types like integers and floating point. So to use those parsing routines, you need to UTF-8 validate to `&str` first. And finally, another nice point about bstr is its `BString` and `BStr` types, which serve as a "target" for trait impls that don't necessarily make sense on a generic `Vec<T>` and `&[T]`. Which is maybe fine for a crate, but it's kind of a clunky solution to the problem. I think if `&str` was conventionally UTF-8, then a lot of this bifurcation and pain wouldn't exist. Because you could just zero-cost convert a `&[u8]` to a `&str` in safe Rust without any hitch. And then you get every method available on `&str` for free given a `&[u8]`.
- camgunz 2y ago
  Yeah that's probably too much of a slog. It's probably the case that very few things would break if you just snuck the validation out the back door and added like u8str (or whatever) that did the validation for people who want that enforced by the type system. Would be real easy to provide migration tooling too.

burntsushi 2y ago

Yeah I think there is some desire for bstr to come into std. But... it's like... tricky. We're steadily adding "string-like" methods to `&[u8]`, but for example, we still lack substring search on `&[u8]`. And lots of other things too. And there are things to consider like the fact that std has this elaborate `Pattern` abstraction (which is itself unstable, but still permits ergonomic polymorphic use of routines that are generic over it). Does that also need to get moved to `&[u8]`? And then there are things like `FromStr` that std implements for lots of types like integers and floating point. So to use those parsing routines, you need to UTF-8 validate to `&str` first. And finally, another nice point about bstr is its `BString` and `BStr` types, which serve as a "target" for trait impls that don't necessarily make sense on a generic `Vec<T>` and `&[T]`. Which is maybe fine for a crate, but it's kind of a clunky solution to the problem. I think if `&str` was conventionally UTF-8, then a lot of this bifurcation and pain wouldn't exist. Because you could just zero-cost convert a `&[u8]` to a `&str` in safe Rust without any hitch. And then you get every method available on `&str` for free given a `&[u8]`.

camgunz 2y ago
Yeah that's probably too much of a slog. It's probably the case that very few things would break if you just snuck the validation out the back door and added like u8str (or whatever) that did the validation for people who want that enforced by the type system. Would be real easy to provide migration tooling too.