4 ms·

Java doesn't read uniboms sanely. Heaven help you parsing mixed input files.

• neonsunset 3y ago
  It must be very difficult to use one of the dozens of packages that let you detect and pick correct encoding to wrap a file stream before reading into a string.
  • bobmcnamara 3y ago
    Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the problem is ambiguous, we had the best luck with a library that allowed us to disable possible encodings. We thought we could just remove encodings that were not widely in use in the countries we served, and that mostly fixed it, but it still came up a couple times a year. Eventually we had a large enough corpus of sample documents that we could test any changes to the character set bitmap for impact to at least some of each better supported character set. Just not my cup of tea though.