6 ms·

>> - You can't just skip stuff if you run any kind of normalization > - delay validation until normalization You probably want to normalize if you want to use

>> - You can't just skip stuff if you run any kind of normalization > - delay validation until normalization You probably want to normalize if you want to use them as keys in a hash table, which is pretty common. Or serialize them to JSON (or protobufs, or msgpack, etc. etc.), also pretty common. Why? Because you're commonly comparing strings and normalization lets strings that should be equal be equal. Canonicalization also requires sorting which requires normalization, etc. etc. >> - How would you index into or split an invalid UTF-8 string? > - treat non-utf-8 bytes as bytes Well I mean by code point, or grapheme. Does the invalid byte go with the 1st or the 2nd? What about multiple invalid bytes? Will all implementations agree? Also hey, looks like you scanned a string and found some invalid bytes. Looks like you validated a UTF-8 string! >> - How would you apply a regex? > - regexes match bytes just fine: the article has a whole section on ripgrep’s use of bstr They definitely don't [0]. If you're allowing invalid UTF-8 you're giving up Unicode support entirely, by necessity. >> - What is its length? > - another section discusses how the length of a string is not a well defined quantity The article doesn't say that, it says you can know neither the printed length of a Unicode string just by its contents, nor the number of characters in a Unicode string. It's right about the first, but wrong about the second: a Unicode character is represented by a Unicode code point, so their number is knowable. UTF-8 strings have a specific number of bytes and a specific number of code points and graphemes. You can know the number of bytes, code points or graphemes, but not if the string is invalid, because all bets are off. >> - How do you deal with other systems that do validate UTF-8 strings? > - the article says you can delay validation until required My general point is there are very few things you can do without validating. If you're passing them around completely internally, never normalizing, never sending to another system, never writing them into a file name or a file itself, etc. You can gain a tiny amount of speedup by not validating, but it's a huge risk and not worth it for 99% of programs. [0]: https://docs.rs/regex/1.10.4/regex/bytes/index.html#syntax https://docs.rs/regex/1.10.4/regex/bytes/index.html#syntax

burntsushi 2y ago
Your link to the regex crate docs doesn't support what you're saying, as far as I can tell. You can search on `&[u8]` just fine with full Unicode support. Invalid UTF-8 just won't match. > My general point is there are very few things you can do without validating. Pretty much all string processing I've built up in Rust over the past ten years is based on the idea of not validating up front. It's too costly to do. There's an entire section in my blog post on bstr about this: https://blog.burntsushi.net/bstr/#motivation-based-on-performance https://blog.burntsushi.net/bstr/#motivation-based-on-perfor... The rest of the blog may be very useful to read as well.
- camgunz 2y ago
  I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." > Pretty much all string processing I've built up in Rust over the past ten years is based on the idea of not validating up front. It's too costly to do. There's an entire section in my blog post on bstr about this: https://blog.burntsushi.net/bstr/#motivation-based-on-perfor https://blog.burntsushi.net/bstr/#motivation-based-on-perfor... Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it.
  - burntsushi 2y ago
    > I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." But that doesn't throw out Unicode support. ASCII compatible mode is literally defined as "Unicode support is disabled and the atom of matching is the individual byte instead of the codepoint." But that only applies to the region of the pattern in which Unicode mode is disabled. Searching on `&[u8]` doesn't automatically enable ASCII compatible mode. It merely makes it possible to enable it in contexts that would otherwise match invalid UTF-8. For example, you cannot search for `(?-u:[^a])` on `&str` because `(?-u:[^a])` matches any individual byte except for the ASCII `a`. That includes things like `\xFF` or even a prefix of a valid UTF-8 encoded Unicode scalar value. But if you search on `&[u8]`, then constructs like `(?-u:[^a])` are allowed. But disabling Unicode mode when searching a `&str` is still allowed, but only if it's still guaranteed to match valid UTF-8. For example: $ echo 'ſ' | rg '(?i-u:s)' $ echo 'ſ' | rg '(?i:s)' ſ But when searching `&[u8]`, Unicode mode is still enabled by default. `rg` searches `&[u8]`, and you can see right above that it's still doing a Unicode aware case insensitive search. > Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it. It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. To be clear, I am not staking a position like the OP where I'm saying "conventionally UTF-8" instead of "required UTF-8" is always better or should be the preferred design in a primitive string data type. My own personal bias is certainly towards that, but as someone who is also a steward of Rust's standard library, I do not limit myself to the world of grep or Unix command line tools. My point here is not that you're wrong about requiring UTF-8 validation as a better design, but rather, that you aren't quite getting the details right when talking about the downsides of conventional UTF-8. I'm not clear on what you mean by "safety trade-offs" or "weirdo lack of specification." The Unicode consortium has defined rules for dealing with invalid sequences of bytes. bstr's API docs have an entire section about this[1], and its behavior even comes from a W3C standard. The difference between "conventional UTF-8" and "required UTF-8" is not really one of safety or lack of specification. In either design, you can expose the same logical semantics. The real differences tend to come in two flavors. First is that when you have a UTF-8 guarantee and your representation is UTF-8, you can do UTF-8 decoding potentially faster than you could otherwise (safely) do because you can eliminate certain conditional branches and error checking. (But this advantage is rendered moot if validation forces you to do two passes over a string that is large.) The second benefit is that it forces a stronger discipline of discarding junk data at the boundaries. It makes failure modes as a result of bad data more explicit. It is this second benefit that I think is the most compelling advantage personally, and why I think, on balance, Rust's design is better. But it's not obviously The Correct Choice. [1]: https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8 https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8
zzo38computer 2y ago
> You probably want to normalize if you want to use them as keys in a hash table, which is pretty common. Not necessarily. Sometimes this might be useful, but sometimes it will just make it worse. > Does the invalid byte go with the 1st or the 2nd? What about multiple invalid bytes? What seems to me the most logical way to do it, if you do not then need to decode it into Unicode code point numbers, is: If the byte is in range 0x80 to 0xBF, then it belongs to the same code point as the previous byte; otherwise, the byte is the start of a new code point. (However, this does not answer the question of what if the first byte is in range 0x80 to 0xBF?) However, that is usually unnecessary. Splitting a string by Unicode code points is rarely helpful (unless you want to convert to UTF-16 or UTF-32) and is often harmful, anyways. Just use sequences of bytes. > if you're allowing invalid UTF-8 you're giving up Unicode support entirely, by necessity. It is often useful to do regular expressions on sequences of bytes instead of on sequences of Unicode code points anyways; and if you have ASCII only, then doing it this way is more efficient anyways (even if the text contains non-ASCII characters). (PCRE has a UTF-8-oriented mode and byte-oriented mode, so both are possible.) Also, you can match a UTF-8 code point (or a character code in EUC-JP or some other multibyte encoding) with a byte-oriented regular expression if you need to, anyways, so still you might not need Unicode regular expressions. > another section discusses how the length of a string is not a well defined quantity I think that the number of bytes is generally the most useful measure, although in some contexts it is useful to measure something else.

burntsushi 2y ago

Your link to the regex crate docs doesn't support what you're saying, as far as I can tell. You can search on `&[u8]` just fine with full Unicode support. Invalid UTF-8 just won't match. > My general point is there are very few things you can do without validating. Pretty much all string processing I've built up in Rust over the past ten years is based on the idea of not validating up front. It's too costly to do. There's an entire section in my blog post on bstr about this: https://blog.burntsushi.net/bstr/#motivation-based-on-performance https://blog.burntsushi.net/bstr/#motivation-based-on-perfor... The rest of the blog may be very useful to read as well.

camgunz 2y ago
I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." > Pretty much all string processing I've built up in Rust over the past ten years is based on the idea of not validating up front. It's too costly to do. There's an entire section in my blog post on bstr about this: https://blog.burntsushi.net/bstr/#motivation-based-on-perfor https://blog.burntsushi.net/bstr/#motivation-based-on-perfor... Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it.
- burntsushi 2y ago
  > I'm referring to #2 there: "In ASCII compatible mode, Unicode character classes are not allowed." But that doesn't throw out Unicode support. ASCII compatible mode is literally defined as "Unicode support is disabled and the atom of matching is the individual byte instead of the codepoint." But that only applies to the region of the pattern in which Unicode mode is disabled. Searching on `&[u8]` doesn't automatically enable ASCII compatible mode. It merely makes it possible to enable it in contexts that would otherwise match invalid UTF-8. For example, you cannot search for `(?-u:[^a])` on `&str` because `(?-u:[^a])` matches any individual byte except for the ASCII `a`. That includes things like `\xFF` or even a prefix of a valid UTF-8 encoded Unicode scalar value. But if you search on `&[u8]`, then constructs like `(?-u:[^a])` are allowed. But disabling Unicode mode when searching a `&str` is still allowed, but only if it's still guaranteed to match valid UTF-8. For example: $ echo 'ſ' | rg '(?i-u:s)' $ echo 'ſ' | rg '(?i:s)' ſ But when searching `&[u8]`, Unicode mode is still enabled by default. `rg` searches `&[u8]`, and you can see right above that it's still doing a Unicode aware case insensitive search. > Yeah I read it; it's very good! (like everything you write) I think you're right that you can't handwave away the validation overhead, and that in some cases it matters. I just think ripgrep is pretty niche: there's gazillions of websites out there and like, 10 greps. If you're writing a grep and you want to optimize the shit out of it, well welcome to working around not validating. Otherwise, the safety tradeoffs and weirdo lack of specification aren't worth it. It's not just grep. It's pretty much any tool that wants to deal with arbitrary file content where it's useful. To be clear, I am not staking a position like the OP where I'm saying "conventionally UTF-8" instead of "required UTF-8" is always better or should be the preferred design in a primitive string data type. My own personal bias is certainly towards that, but as someone who is also a steward of Rust's standard library, I do not limit myself to the world of grep or Unix command line tools. My point here is not that you're wrong about requiring UTF-8 validation as a better design, but rather, that you aren't quite getting the details right when talking about the downsides of conventional UTF-8. I'm not clear on what you mean by "safety trade-offs" or "weirdo lack of specification." The Unicode consortium has defined rules for dealing with invalid sequences of bytes. bstr's API docs have an entire section about this[1], and its behavior even comes from a W3C standard. The difference between "conventional UTF-8" and "required UTF-8" is not really one of safety or lack of specification. In either design, you can expose the same logical semantics. The real differences tend to come in two flavors. First is that when you have a UTF-8 guarantee and your representation is UTF-8, you can do UTF-8 decoding potentially faster than you could otherwise (safely) do because you can eliminate certain conditional branches and error checking. (But this advantage is rendered moot if validation forces you to do two passes over a string that is large.) The second benefit is that it forces a stronger discipline of discarding junk data at the boundaries. It makes failure modes as a result of bad data more explicit. It is this second benefit that I think is the most compelling advantage personally, and why I think, on balance, Rust's design is better. But it's not obviously The Correct Choice. [1]: https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8 https://docs.rs/bstr/latest/bstr/#handling-of-invalid-utf-8

zzo38computer 2y ago

> You probably want to normalize if you want to use them as keys in a hash table, which is pretty common. Not necessarily. Sometimes this might be useful, but sometimes it will just make it worse. > Does the invalid byte go with the 1st or the 2nd? What about multiple invalid bytes? What seems to me the most logical way to do it, if you do not then need to decode it into Unicode code point numbers, is: If the byte is in range 0x80 to 0xBF, then it belongs to the same code point as the previous byte; otherwise, the byte is the start of a new code point. (However, this does not answer the question of what if the first byte is in range 0x80 to 0xBF?) However, that is usually unnecessary. Splitting a string by Unicode code points is rarely helpful (unless you want to convert to UTF-16 or UTF-32) and is often harmful, anyways. Just use sequences of bytes. > if you're allowing invalid UTF-8 you're giving up Unicode support entirely, by necessity. It is often useful to do regular expressions on sequences of bytes instead of on sequences of Unicode code points anyways; and if you have ASCII only, then doing it this way is more efficient anyways (even if the text contains non-ASCII characters). (PCRE has a UTF-8-oriented mode and byte-oriented mode, so both are possible.) Also, you can match a UTF-8 code point (or a character code in EUC-JP or some other multibyte encoding) with a byte-oriented regular expression if you need to, anyways, so still you might not need Unicode regular expressions. > another section discusses how the length of a string is not a well defined quantity I think that the number of bytes is generally the most useful measure, although in some contexts it is useful to measure something else.