4 ms·

Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime

Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime library in Rust. One of the main features of a datetime library is the ability to parse datetimes out of strings. The standard way to do this in Rust is with the `FromStr` trait, which requires that you pass in a `&str`. But there's really nothing about datetime parsing that requires a `&str`. And so if you just do what the "standard" thing in Rust is (use a `&str`), then you're automatically requiring that every single datetime parse go through a separate UTF-8 validation step first, which is required in order to get a `&str`. This puts an unavoidable hit on perf that isn't actually necessary for the task at hand. Imagine for example parsing a CSV file. The `csv` crate pushes you toward using `&str` for everything, but you can also parse CSV data as fields of `&[u8]` instead. Skipping that UTF-8 validation will most definitely speed things up. Now imagine you want to parse a datetime from a field. Is it better to just parse your `&[u8]` field directly, or should I make you do UTF-8 validation to get a `&str` first? IMO, the correct answer here is to define a way to parse datetimes out of a `&[u8]`, while pushing folks in the API toward using `&str` for "most" cases. The hitch is that if you want to parse with `&[u8]`, you're now working outside the domain of standard Rust strings. And things become just a little trickier. It's not the paved path that `&str` is. And this all comes from the fact that Rust strings required UTF-8. The great thing about parsing `&[u8]` is that you can reuse that implementation for parsing from `&str` too. But you really do have to know to do the implementation on `&[u8]` first. Unless you know all the nuances of required UTF-8 versus conventional UTF-8, it's not obvious. Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-)

camgunz 2y ago
I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, often commits the sin of requiring unnecessary processing. Your CSV example here is a pretty good one: you either need heuristics or some kind of OOB data to "know" if you've got an encoding that can possibly represent a CSV file. If you've got that, you can always at least convert to a compatible encoding, e.g. UTF-8. If you don't, you can either bail or cross your fingers and hope that it'll work out (this is perfectly reasonable in many cases). Regardless, if you want to give programmers all of these options with zero cost you need a fully functional encoding-agnostic byte string type. I do think you've done the legwork to improve the situation though? It would be cool if Rust incorporated bstr into std; I think that would give library authors (and maybe std authors) confidence in working on a lower level interface that didn't guarantee/require valid UTF-8. > Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-) 100% agree. Honestly, re-reading OP it's clear I got really hung up on the bullet points in the middle ("Do you know how to render it" etc.), but it is really about UTF-8 in programming language string types. And hey, empirically it worked haha :)
- burntsushi 2y ago
  Yeah I think there is some desire for bstr to come into std. But... it's like... tricky. We're steadily adding "string-like" methods to `&[u8]`, but for example, we still lack substring search on `&[u8]`. And lots of other things too. And there are things to consider like the fact that std has this elaborate `Pattern` abstraction (which is itself unstable, but still permits ergonomic polymorphic use of routines that are generic over it). Does that also need to get moved to `&[u8]`? And then there are things like `FromStr` that std implements for lots of types like integers and floating point. So to use those parsing routines, you need to UTF-8 validate to `&str` first. And finally, another nice point about bstr is its `BString` and `BStr` types, which serve as a "target" for trait impls that don't necessarily make sense on a generic `Vec<T>` and `&[T]`. Which is maybe fine for a crate, but it's kind of a clunky solution to the problem. I think if `&str` was conventionally UTF-8, then a lot of this bifurcation and pain wouldn't exist. Because you could just zero-cost convert a `&[u8]` to a `&str` in safe Rust without any hitch. And then you get every method available on `&str` for free given a `&[u8]`.
  - camgunz 2y ago
    Yeah that's probably too much of a slog. It's probably the case that very few things would break if you just snuck the validation out the back door and added like u8str (or whatever) that did the validation for people who want that enforced by the type system. Would be real easy to provide migration tooling too.

camgunz 2y ago

I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, often commits the sin of requiring unnecessary processing. Your CSV example here is a pretty good one: you either need heuristics or some kind of OOB data to "know" if you've got an encoding that can possibly represent a CSV file. If you've got that, you can always at least convert to a compatible encoding, e.g. UTF-8. If you don't, you can either bail or cross your fingers and hope that it'll work out (this is perfectly reasonable in many cases). Regardless, if you want to give programmers all of these options with zero cost you need a fully functional encoding-agnostic byte string type. I do think you've done the legwork to improve the situation though? It would be cool if Rust incorporated bstr into std; I think that would give library authors (and maybe std authors) confidence in working on a lower level interface that didn't guarantee/require valid UTF-8. > Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-) 100% agree. Honestly, re-reading OP it's clear I got really hung up on the bullet points in the middle ("Do you know how to render it" etc.), but it is really about UTF-8 in programming language string types. And hey, empirically it worked haha :)

burntsushi 2y ago
Yeah I think there is some desire for bstr to come into std. But... it's like... tricky. We're steadily adding "string-like" methods to `&[u8]`, but for example, we still lack substring search on `&[u8]`. And lots of other things too. And there are things to consider like the fact that std has this elaborate `Pattern` abstraction (which is itself unstable, but still permits ergonomic polymorphic use of routines that are generic over it). Does that also need to get moved to `&[u8]`? And then there are things like `FromStr` that std implements for lots of types like integers and floating point. So to use those parsing routines, you need to UTF-8 validate to `&str` first. And finally, another nice point about bstr is its `BString` and `BStr` types, which serve as a "target" for trait impls that don't necessarily make sense on a generic `Vec<T>` and `&[T]`. Which is maybe fine for a crate, but it's kind of a clunky solution to the problem. I think if `&str` was conventionally UTF-8, then a lot of this bifurcation and pain wouldn't exist. Because you could just zero-cost convert a `&[u8]` to a `&str` in safe Rust without any hitch. And then you get every method available on `&str` for free given a `&[u8]`.
- camgunz 2y ago
  Yeah that's probably too much of a slog. It's probably the case that very few things would break if you just snuck the validation out the back door and added like u8str (or whatever) that did the validation for people who want that enforced by the type system. Would be real easy to provide migration tooling too.