3 ms·
Afaik Rust (currently) uses WTF-8 [0] to store Windows (WTF-16) strings which is a very useful encoding if you need to deal with such strings in your programs w
by account42 3y ago
Afaik Rust (currently) uses WTF-8 [0] to store Windows (WTF-16) strings which is a very useful encoding if you need to deal with such strings in your programs written in other languages as well. The conversion is essentially the same as UTF-16 -> to UTF-8 except you interpret unmatched surrogate pairs as (reserved) Unicode values with the same value and encode those to UTF-8 as you would any other Unicode value. So this doesn't use exactly the same trick as TFA - instead of using invalid UTF-8 encodings it uses "normal" UTF-8 encodings of invalid Unicode values (specifically exactly those that were reserved when extending UCS-2 to UTF-16). Or in other words, WTF-16 <-> WTF-8 conversion is the same as UTF-16 <-> UTF-8 conversion but without (some of) the error handling.
[0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/