3 ms·
Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the pro
by bobmcnamara 3y ago
Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the problem is ambiguous, we had the best luck with a library that allowed us to disable possible encodings. We thought we could just remove encodings that were not widely in use in the countries we served, and that mostly fixed it, but it still came up a couple times a year. Eventually we had a large enough corpus of sample documents that we could test any changes to the character set bitmap for impact to at least some of each better supported character set.
Just not my cup of tea though.