3 ms·
Probably it happens when the real text encoding is different from the declared one (at least that's the problem I encountered when aggregating content from unfi
by bd 18y ago
Probably it happens when the real text encoding is different from the declared one (at least that's the problem I encountered when aggregating content from unfiltered wild web).
If it really bothers, you can use chardet [1] to try to detect the real encoding (BeautifulSoup should use it if it's installed). But even this is not 100% foolproof.
[1] http://chardet.feedparser.org/ http://chardet.feedparser.org/
- jauco 18y agoYep, ’ means that the pages contains the UTF-8 quote character, but the browser renders the bytestream as if it's a single byte character stream. The crux is that the basic alphabet is encoded the same. So you only notice it with special characters such as the curly quote and the em-dash.