3 ms·
All good points. However, there are more issues with dealing with strings in the wild than just round-tripping between encodings. Many of the issues I've dealt
by wooster 17y ago
All good points. However, there are more issues with dealing with strings in the wild than just round-tripping between encodings.
Many of the issues I've dealt with when data mining involved mixed encodings within the same document, documents labelled with the wrong encoding in the metadata, and documents with no encoding information. There's only so much you can do as far as sniffing character sets and languages to avoid mojibake and other, more subtle, problems.
For my purposes, converting to Unicode and losing round-tripping is only a minor concern, whereas dealing with non-Unicode encodings is often a source of major problems.
So, personally, having worked both in languages that deal with strings by converting them internally to Unicode, and ones that treat them as encoding-tagged byte streams, I definitely favor the ones that deal with them as Unicode. But, my purposes aren't everyone's, and I'm not convinced there's a paradigm that would suit both usage patterns.