3 ms·
I think I would point to Go here as a good data point for mollification. Its `string` data is conventionally UTF-8. Go has been a huge player in web services ov
by burntsushi 2y ago
I think I would point to Go here as a good data point for mollification. Its `string` data is conventionally UTF-8. Go has been a huge player in web services over the last decade and a half, and I'm not sure I've seen a rise in the problem of junk data.
The people behind Go were also the people behind UTF-8 (Ken Thompson and Rob Pike). Go's `string` data type is thoughtfully designed. If you iterate over the characters in a string that contains invalid UTF-8, you get replacement codepoints, just like in bstr. Go has transcoding and normalization libraries that work just fine too.
I don't think there are much if any weird interactions with normalization or other such things. When you're doing Unicode operations on a conventionally UTF-8 string, you just treat invalid UTF-8 as U+FFFD. That's really all there is to it. Everything else works. Unicode algorithms are defined over codepoints. So all you need is a semantic for how to convert invalid sequences to codepoints. And Unicode (Chapter 3, Section 9) provides exactly that for you.
Or in some cases, you can just leave the invalid UTF-8 as-is. The last example in bstr's docs for its "to_lowercase" routine[1], for example, demonstrates how you can perform a Unicode-aware algorithm on a string that contains invalid UTF-8.
We've all been burned by mojibake before. I get it. I'm not totally discounting your viewpoint. I do tend to think that "required UTF-8" is probably the better thing for a primitive string data type. But you really do absolutely need to also then support conventional UTF-8. Otherwise you're cutting off a ton of very valid use cases, and potentially forcing others to either take a huge perf hit or to do something dangerous like `unsafe { std::str::from_utf8_unchecked(..) }`. I've seen people do the unchecked variant before just because they had a `&[u8]` but needed it to be a `&str` because they wanted to treat it as a string without doing a second pass over the data. And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-)
[1]: https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#examples-58 https://docs.rs/bstr/latest/bstr/trait.ByteSlice.html#exampl...
- camgunz 2y agoThe Go thing (+ invalid sequence conversion) is a really good point. If OP's point was "don't validate just convert invalid sequences when you need to" or "think twice before forcing validation into your primitive string type" I'd buy in. I also think your choice there to just write the invalid bytes out unchanged also makes a ton of sense. You're right that there's more options than "yolo no validate" or "validate or face death". > And believe you me, you'd see a lot more of that if I wasn't so careful to provide both `&str` and `&[u8]` APIs for all string oriented operations. ;-) Woof man yeah, that's what I was getting at with you're forcing people to basically implement things twice; it's not the best! This was a really good conversation; before this I was 100% in agreement w/ you on "'required UTF-8' is probably the better thing for a primitive string type" but now I see there's a fair amount of pragmatic issues with that. Definitely a lot to mull over here; thanks :)