2 ms·

> But that only applies to the region of the pattern in which Unicode mode is disabled. Yeah I mean, fair but I'm not impugning bstr or byte-based regexes. Bro

> But that only applies to the region of the pattern in which Unicode mode is disabled. Yeah I mean, fair but I'm not impugning bstr or byte-based regexes. Broadly my worry here is people will say, "I'll be cool; I'll use &[u8] everywhere and skip validation" and then run into weirdness when using regexes because the behavior is subtly different, or some features are missing, especially when other stuff is in the mix like invalid UTF-8 or not normalizing. I guess I can file it under "if you're using &[u8] regexes you probably know what you're doing" though. > It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. I do think there's a broader discussion to be had about this like, is it The Correct Choice for a language to enforce an encoding (and implicitly validation) in their primitive string type? Most languages I'd say yeah; it's the kind of thing you should abstract away, the safety benefits outweigh the performance costs, and you can always provide some kind of trapdoor (Python's BytesIO or w/e, etc.). I'm not sure about systems languages; feels you're at least asking everything that deals with (let's just say) &str to also deal with &[u8] in order to preserve choice, which is at least tedious for library authors. Dunno what the right mechanism here is but, I'm definitely sympathetic. > you aren't quite getting the details right when talking about the downsides of conventional UTF-8 Ha well that's certainly possible :) I might be throwing "safety" around too carelessly; really what I'm saying here is "if you validate, your program might be a very small bit slower but you'll avoid spreading the virus of invalid UTF-8 through your system and others'" There are a lot of benefits to that -- a lot of other programs will still validate by default, so if they read output from a program that isn't validating, they might choke or behave in subtly bad ways leading to all kinds of problems ranging from junk on the screen to some kind of security bypass because there's now invalid UTF-8 in a JWT. That's bad! Why would you risk it? You described the benefits really well there (I've written a Unicode-aware string library in C and come to the same conclusions) but, again I'm sympathetic to other totally valid use cases where you're like, "but my data is in UTF-16, a perfectly valid format that's very very popular, and you want me to convert it _every time_?" Tough to take, again especially if you're dealing with lots of data.

burntsushi 2y ago
Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime library in Rust. One of the main features of a datetime library is the ability to parse datetimes out of strings. The standard way to do this in Rust is with the `FromStr` trait, which requires that you pass in a `&str`. But there's really nothing about datetime parsing that requires a `&str`. And so if you just do what the "standard" thing in Rust is (use a `&str`), then you're automatically requiring that every single datetime parse go through a separate UTF-8 validation step first, which is required in order to get a `&str`. This puts an unavoidable hit on perf that isn't actually necessary for the task at hand. Imagine for example parsing a CSV file. The `csv` crate pushes you toward using `&str` for everything, but you can also parse CSV data as fields of `&[u8]` instead. Skipping that UTF-8 validation will most definitely speed things up. Now imagine you want to parse a datetime from a field. Is it better to just parse your `&[u8]` field directly, or should I make you do UTF-8 validation to get a `&str` first? IMO, the correct answer here is to define a way to parse datetimes out of a `&[u8]`, while pushing folks in the API toward using `&str` for "most" cases. The hitch is that if you want to parse with `&[u8]`, you're now working outside the domain of standard Rust strings. And things become just a little trickier. It's not the paved path that `&str` is. And this all comes from the fact that Rust strings required UTF-8. The great thing about parsing `&[u8]` is that you can reuse that implementation for parsing from `&str` too. But you really do have to know to do the implementation on `&[u8]` first. Unless you know all the nuances of required UTF-8 versus conventional UTF-8, it's not obvious. Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-)
- camgunz 2y ago
  I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, often commits the sin of requiring unnecessary processing. Your CSV example here is a pretty good one: you either need heuristics or some kind of OOB data to "know" if you've got an encoding that can possibly represent a CSV file. If you've got that, you can always at least convert to a compatible encoding, e.g. UTF-8. If you don't, you can either bail or cross your fingers and hope that it'll work out (this is perfectly reasonable in many cases). Regardless, if you want to give programmers all of these options with zero cost you need a fully functional encoding-agnostic byte string type. I do think you've done the legwork to improve the situation though? It would be cool if Rust incorporated bstr into std; I think that would give library authors (and maybe std authors) confidence in working on a lower level interface that didn't guarantee/require valid UTF-8. > Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-) 100% agree. Honestly, re-reading OP it's clear I got really hung up on the bullet points in the middle ("Do you know how to render it" etc.), but it is really about UTF-8 in programming language string types. And hey, empirically it worked haha :)

burntsushi 2y ago

Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime library in Rust. One of the main features of a datetime library is the ability to parse datetimes out of strings. The standard way to do this in Rust is with the `FromStr` trait, which requires that you pass in a `&str`. But there's really nothing about datetime parsing that requires a `&str`. And so if you just do what the "standard" thing in Rust is (use a `&str`), then you're automatically requiring that every single datetime parse go through a separate UTF-8 validation step first, which is required in order to get a `&str`. This puts an unavoidable hit on perf that isn't actually necessary for the task at hand. Imagine for example parsing a CSV file. The `csv` crate pushes you toward using `&str` for everything, but you can also parse CSV data as fields of `&[u8]` instead. Skipping that UTF-8 validation will most definitely speed things up. Now imagine you want to parse a datetime from a field. Is it better to just parse your `&[u8]` field directly, or should I make you do UTF-8 validation to get a `&str` first? IMO, the correct answer here is to define a way to parse datetimes out of a `&[u8]`, while pushing folks in the API toward using `&str` for "most" cases. The hitch is that if you want to parse with `&[u8]`, you're now working outside the domain of standard Rust strings. And things become just a little trickier. It's not the paved path that `&str` is. And this all comes from the fact that Rust strings required UTF-8. The great thing about parsing `&[u8]` is that you can reuse that implementation for parsing from `&str` too. But you really do have to know to do the implementation on `&[u8]` first. Unless you know all the nuances of required UTF-8 versus conventional UTF-8, it's not obvious. Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-)

camgunz 2y ago
I'm pretty well convinced at this point that specifying an encoding for a system language's string type is probably too constraining--and like you point out, often commits the sin of requiring unnecessary processing. Your CSV example here is a pretty good one: you either need heuristics or some kind of OOB data to "know" if you've got an encoding that can possibly represent a CSV file. If you've got that, you can always at least convert to a compatible encoding, e.g. UTF-8. If you don't, you can either bail or cross your fingers and hope that it'll work out (this is perfectly reasonable in many cases). Regardless, if you want to give programmers all of these options with zero cost you need a fully functional encoding-agnostic byte string type. I do think you've done the legwork to improve the situation though? It would be cool if Rust incorporated bstr into std; I think that would give library authors (and maybe std authors) confidence in working on a lower level interface that didn't guarantee/require valid UTF-8. > Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-) 100% agree. Honestly, re-reading OP it's clear I got really hung up on the bullet points in the middle ("Do you know how to render it" etc.), but it is really about UTF-8 in programming language string types. And hey, empirically it worked haha :)