3 ms·
Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small numb
by duskwuff 3y ago
Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
- ranger_danger 3y agoI don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
- kelnos 3y agoEven then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.
- ranger_danger 3y agoI disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.
- duskwuff 3y agoEncoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
- ranger_danger 3y agoTaking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.