Is that really necessary? Web browsers do no statistically analysis, and yet users seem to have no problem with them? Okay, certainly they have menus which allo

Is that really necessary? Web browsers do no statistically analysis, and yet users seem to have no problem with them? Okay, certainly they have menus which allow you to override them, but they are relatively infrequently used. The most common issue with web scrapers that I've seen is they don't detect the character encoding the same way as web browsers do — which is what web authors expect. http://www.whatwg.org/C#determining-the-character-encoding http://www.whatwg.org/C#determining-the-character-encoding is what the spec defines, and though the eventual fallback is implementation-defined (Firefox, for example, combines locale-specific defaults with TLD-based defaults).