4 ms·

> I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." But that doesn't throw out Unicode support. ASCII compatibl

> I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." But that doesn't throw out Unicode support. ASCII compatible mode is literally defined as "Unicode support is disabled and the atom of matching is the individual byte instead of the codepoint." But that only applies to the region of the pattern in which Unicode mode is disabled. Searching on `&[u8]` doesn't automatically enable ASCII compatible mode. It merely makes it possible to enable it in contexts that would otherwise match invalid UTF-8. For example, you cannot search for `(?-u:[^a])` on `&str` because `(?-u:[^a])` matches any individual byte except for the ASCII `a`. That includes things like `\xFF` or even a prefix of a valid UTF-8 encoded Unicode scalar value. But if you search on `&[u8]`, then constructs like `(?-u:[^a])` are allowed. But disabling Unicode mode when searching a `&str` is still allowed, but only if it's still guaranteed to match valid UTF-8. For example: $ echo 'ſ' | rg '(?i-u:s)' $ echo 'ſ' | rg '(?i:s)' ſ But when searching `&[u8]`, Unicode mode is still enabled by default. `rg` searches `&[u8]`, and you can see right above that it's still doing a Unicode aware case insensitive search. > Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it. It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. To be clear, I am not staking a position like the OP where I'm saying "conventionally UTF-8" instead of "required UTF-8" is always better or should be the preferred design in a primitive string data type. My own personal bias is certainly towards that, but as someone who is also a steward of Rust's standard library, I do not limit myself to the world of grep or Unix command line tools. My point here is not that you're wrong about requiring UTF-8 validation as a better design, but rather, that you aren't quite getting the details right when talking about the downsides of conventional UTF-8. I'm not clear on what you mean by "safety trade-offs" or "weirdo lack of specification." The Unicode consortium has defined rules for dealing with invalid sequences of bytes. bstr's API docs have an entire section about this[1], and its behavior even comes from a W3C standard. The difference between "conventional UTF-8" and "required UTF-8" is not really one of safety or lack of specification. In either design, you can expose the same logical semantics. The real differences tend to come in two flavors. First is that when you have a UTF-8 guarantee and your representation is UTF-8, you can do UTF-8 decoding potentially faster than you could otherwise (safely) do because you can eliminate certain conditional branches and error checking. (But this advantage is rendered moot if validation forces you to do two passes over a string that is large.) The second benefit is that it forces a stronger discipline of discarding junk data at the boundaries. It makes failure modes as a result of bad data more explicit. It is this second benefit that I think is the most compelling advantage personally, and why I think, on balance, Rust's design is better. But it's not obviously The Correct Choice. [1]: https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8 https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8

camgunz 2y ago
> But that only applies to the region of the pattern in which Unicode mode is disabled. Yeah I mean, fair but I'm not impugning bstr or byte-based regexes. Broadly my worry here is people will say, "I'll be cool; I'll use &[u8] everywhere and skip validation" and then run into weirdness when using regexes because the behavior is subtly different, or some features are missing, especially when other stuff is in the mix like invalid UTF-8 or not normalizing. I guess I can file it under "if you're using &[u8] regexes you probably know what you're doing" though. > It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. I do think there's a broader discussion to be had about this like, is it The Correct Choice for a language to enforce an encoding (and implicitly validation) in their primitive string type? Most languages I'd say yeah; it's the kind of thing you should abstract away, the safety benefits outweigh the performance costs, and you can always provide some kind of trapdoor (Python's BytesIO or w/e, etc.). I'm not sure about systems languages; feels you're at least asking everything that deals with (let's just say) &str to also deal with &[u8] in order to preserve choice, which is at least tedious for library authors. Dunno what the right mechanism here is but, I'm definitely sympathetic. > you aren't quite getting the details right when talking about the downsides of conventional UTF-8 Ha well that's certainly possible :) I might be throwing "safety" around too carelessly; really what I'm saying here is "if you validate, your program might be a very small bit slower but you'll avoid spreading the virus of invalid UTF-8 through your system and others'" There are a lot of benefits to that -- a lot of other programs will still validate by default, so if they read output from a program that isn't validating, they might choke or behave in subtly bad ways leading to all kinds of problems ranging from junk on the screen to some kind of security bypass because there's now invalid UTF-8 in a JWT. That's bad! Why would you risk it? You described the benefits really well there (I've written a Unicode-aware string library in C and come to the same conclusions) but, again I'm sympathetic to other totally valid use cases where you're like, "but my data is in UTF-16, a perfectly valid format that's very very popular, and you want me to convert it _every time_?" Tough to take, again especially if you're dealing with lots of data.
- burntsushi 2y ago
  Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime library in Rust. One of the main features of a datetime library is the ability to parse datetimes out of strings. The standard way to do this in Rust is with the `FromStr` trait, which requires that you pass in a `&str`. But there's really nothing about datetime parsing that requires a `&str`. And so if you just do what the "standard" thing in Rust is (use a `&str`), then you're automatically requiring that every single datetime parse go through a separate UTF-8 validation step first, which is required in order to get a `&str`. This puts an unavoidable hit on perf that isn't actually necessary for the task at hand. Imagine for example parsing a CSV file. The `csv` crate pushes you toward using `&str` for everything, but you can also parse CSV data as fields of `&[u8]` instead. Skipping that UTF-8 validation will most definitely speed things up. Now imagine you want to parse a datetime from a field. Is it better to just parse your `&[u8]` field directly, or should I make you do UTF-8 validation to get a `&str` first? IMO, the correct answer here is to define a way to parse datetimes out of a `&[u8]`, while pushing folks in the API toward using `&str` for "most" cases. The hitch is that if you want to parse with `&[u8]`, you're now working outside the domain of standard Rust strings. And things become just a little trickier. It's not the paved path that `&str` is. And this all comes from the fact that Rust strings required UTF-8. The great thing about parsing `&[u8]` is that you can reuse that implementation for parsing from `&str` too. But you really do have to know to do the implementation on `&[u8]` first. Unless you know all the nuances of required UTF-8 versus conventional UTF-8, it's not obvious. Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-)

camgunz 2y ago

> But that only applies to the region of the pattern in which Unicode mode is disabled. Yeah I mean, fair but I'm not impugning bstr or byte-based regexes. Broadly my worry here is people will say, "I'll be cool; I'll use &[u8] everywhere and skip validation" and then run into weirdness when using regexes because the behavior is subtly different, or some features are missing, especially when other stuff is in the mix like invalid UTF-8 or not normalizing. I guess I can file it under "if you're using &[u8] regexes you probably know what you're doing" though. > It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. I do think there's a broader discussion to be had about this like, is it The Correct Choice for a language to enforce an encoding (and implicitly validation) in their primitive string type? Most languages I'd say yeah; it's the kind of thing you should abstract away, the safety benefits outweigh the performance costs, and you can always provide some kind of trapdoor (Python's BytesIO or w/e, etc.). I'm not sure about systems languages; feels you're at least asking everything that deals with (let's just say) &str to also deal with &[u8] in order to preserve choice, which is at least tedious for library authors. Dunno what the right mechanism here is but, I'm definitely sympathetic. > you aren't quite getting the details right when talking about the downsides of conventional UTF-8 Ha well that's certainly possible :) I might be throwing "safety" around too carelessly; really what I'm saying here is "if you validate, your program might be a very small bit slower but you'll avoid spreading the virus of invalid UTF-8 through your system and others'" There are a lot of benefits to that -- a lot of other programs will still validate by default, so if they read output from a program that isn't validating, they might choke or behave in subtly bad ways leading to all kinds of problems ranging from junk on the screen to some kind of security bypass because there's now invalid UTF-8 in a JWT. That's bad! Why would you risk it? You described the benefits really well there (I've written a Unicode-aware string library in C and come to the same conclusions) but, again I'm sympathetic to other totally valid use cases where you're like, "but my data is in UTF-16, a perfectly valid format that's very very popular, and you want me to convert it _every time_?" Tough to take, again especially if you're dealing with lots of data.

burntsushi 2y ago
Aye. Here's another example. This isn't necessarily responding to any specific point in your comment, but something to mull over. I'm working on a new datetime library in Rust. One of the main features of a datetime library is the ability to parse datetimes out of strings. The standard way to do this in Rust is with the `FromStr` trait, which requires that you pass in a `&str`. But there's really nothing about datetime parsing that requires a `&str`. And so if you just do what the "standard" thing in Rust is (use a `&str`), then you're automatically requiring that every single datetime parse go through a separate UTF-8 validation step first, which is required in order to get a `&str`. This puts an unavoidable hit on perf that isn't actually necessary for the task at hand. Imagine for example parsing a CSV file. The `csv` crate pushes you toward using `&str` for everything, but you can also parse CSV data as fields of `&[u8]` instead. Skipping that UTF-8 validation will most definitely speed things up. Now imagine you want to parse a datetime from a field. Is it better to just parse your `&[u8]` field directly, or should I make you do UTF-8 validation to get a `&str` first? IMO, the correct answer here is to define a way to parse datetimes out of a `&[u8]`, while pushing folks in the API toward using `&str` for "most" cases. The hitch is that if you want to parse with `&[u8]`, you're now working outside the domain of standard Rust strings. And things become just a little trickier. It's not the paved path that `&str` is. And this all comes from the fact that Rust strings required UTF-8. The great thing about parsing `&[u8]` is that you can reuse that implementation for parsing from `&str` too. But you really do have to know to do the implementation on `&[u8]` first. Unless you know all the nuances of required UTF-8 versus conventional UTF-8, it's not obvious. Because of course, the standard party line is, "use strings and strings are always UTF-8, why use anything else?" That question is I think where the OP lives, and getting more folks to think in a nuanced way about this topic is probably a good thing. :-)