3 ms·
"You must provide it clean (interchange-valid) UTF-8, so any encoding issues must be sorted out before-hand." In most cases you have to know the language in or
by sick-boy 15y ago
"You must provide it clean (interchange-valid) UTF-8, so any encoding issues must be sorted out before-hand."
In most cases you have to know the language in order to guess the encoding and convert to UTF-8 if necessary.
Mutual recursion...
- Maakuth 15y agoI'd say in most cases UTF-8 is already used. Of course this depends on the source of the text.
- adambyrtek 15y agoEncoding, but obviously not language, should be provided explicitly as metadata (e.g. Content-Type HTTP header). Also, most of content available on the web is already UTF-8 (65.9% according to a recent survey[1]). [1] http://w3techs.com/technologies/details/en-utf8/all/all http://w3techs.com/technologies/details/en-utf8/all/all
- ninjin 15y agoMark Pilgrim reversed (or ripped out, can't remember) the encoding detection that Firefox uses. It has done a fairly good job for my web crawling: http://pypi.python.org/pypi/chardet http://pypi.python.org/pypi/chardet
- e98cuenc 15y agoIn my experience chardet misclassifies very often iso-8859-1 as iso-8859-2. I saw the misclassification even in small spanish pages, which were using only the typical spanish characters.