3 ms·
I've previously struggled with the encoding of books on Project Gutenberg. In theory each text file has a header that specifies the encoding and other data abou
by yorwba 8y ago
I've previously struggled with the encoding of books on Project Gutenberg. In theory each text file has a header that specifies the encoding and other data about the book like author and title. In practice the header format varies unpredictably (likely it was written manually and not meant for machine consumption) and even if you can parse the encoding value, you might have one that says ASCII but is actually some Windows code page, or UTF-8, or something completely different. And that encoding is also used in other parts of the header, so it can happen that you fail to parse the header because you don't know which encoding to use. One book actually used different encodings in different parts.
After collecting a messy tangle of special cases over several days, I threw in the towel and just used BeautifulSoup's UnicodeDammit to brute-force a working encoding.
Maybe the Swedish book didn't indicate a vs å in the first place, though.
- jfk13 8y agoNot sure this one came from Gutenberg; at least, I can't seem to find a Swedish version of it there. But anyhow, either the source is worthless, or some conversion process somewhere was bad. A vs Å is not a minor or optional issue in Swedish; å, ä and ö are separate letters of the alphabet in their own right, and "omitting accents" isn't a thing. It's just wrong. (E.g. see the Swedish alphabet at http://omniglot.com/writing/swedish.htm http://omniglot.com/writing/swedish.htm) > I threw in the towel and just used BeautifulSoup's UnicodeDammit to brute-force a working encoding. For a limited definition of "working", at times! "A working encoding" that shows the wrong letters isn't "working" in a very useful sense.