10 ms·
The advice about utf-8 encoding is a bit incomplete. While adding <meta charset="utf-8" /> is a good idea, it's also important to make sure that the webserver i
by greglindahl 8y ago
The advice about utf-8 encoding is a bit incomplete. While adding <meta charset="utf-8" /> is a good idea, it's also important to make sure that the webserver isn't sending an http Content-Type header that's something other than utf-8.
In HTML5, the browser is supposed to not sniff the document for a meta charset if the server headers specify a charset in the Content-type.
- kevin_thibedeau 8y agoI've been scraping some Japanese sites lately and this has been a minor annoyance. Content-Type rarely has the encoding and Requests doesn't default to UTF-8 so you get mojibake for EUC-JP and UTF-8 unless you intervene.