3 ms·

My understanding is that Rust has designed the rest of the String API under the assumption of validity. You can't create an invalid String because the methods t

My understanding is that Rust has designed the rest of the String API under the assumption of validity. You can't create an invalid String because the methods that operate on strings strive to be tightly optimized UTF-8 manipulation algorithms that assume the string has already been cleaned. Pack all of the actual robustness in the guarantee that you are working with UTF-8, and you can avoid unnecessary CPU cycles, which is one of the goals of systems languages. If you want to skip it, go for raw strings or CStr -- all raw byte buffers have the basic ASCII functions available, which are designed to be robust against whatever you throw at it, and it shouldn't be too hard to introduce genericity for an API to accept both strings and raw data. That being said, I'm not sure how this is actually implemented, I assume there is still some degree of robustness when running methods on strings generated using `unsafe fn from_utf8_unchecked` just by nature of UTF-8's self-synchronization, which may be what the article is pointing out. It's possible that some cleverly optimized UTF-8 algorithms don't need valid data to avoid memory issues / UB that trips the execution of the entire program, and can instead catch the error or perform a lossy transformation on the spot without incurring too much overhead.

csande17 2y ago
Rust's "all strings must be valid UTF-8" rule allows them to remove a bounds check in their "get the Unicode code point corresponding to a byte index in the string" function. Like, if you see a byte in the string that indicates it's the start of a 3-byte character, you don't have to worry about whether you'll have a buffer overrun when you try to read the following two bytes. As far as I know that's pretty much the only memory-safety-related benefit of that rule, and it's probably a wash since you still need to do that check at some other point in your program to safely construct a &str from non-constant string data. The impression I get from Rust people is that UTF-8 str is mostly about purity for purity's sake, and any performance improvement is at best a minor side benefit.

csande17 2y ago

Rust's "all strings must be valid UTF-8" rule allows them to remove a bounds check in their "get the Unicode code point corresponding to a byte index in the string" function. Like, if you see a byte in the string that indicates it's the start of a 3-byte character, you don't have to worry about whether you'll have a buffer overrun when you try to read the following two bytes. As far as I know that's pretty much the only memory-safety-related benefit of that rule, and it's probably a wash since you still need to do that check at some other point in your program to safely construct a &str from non-constant string data. The impression I get from Rust people is that UTF-8 str is mostly about purity for purity's sake, and any performance improvement is at best a minor side benefit.