3 ms·
Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's
by duskwuff 3y ago
Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
- ranger_danger 3y agoTaking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.