A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding. This question was not relevant for Scrapy be

A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding. This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do when page encoding is unknown; unlike encoding of URLs extracted from HTML this UTF-8 default doesn't lead to incompatible behavior.