4 ms·

You can't assume it if you're handed a random sample of text files.

duskwuff 3y ago
You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
- ranger_danger 3y ago
  Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
  - duskwuff 3y ago
    Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
    - ranger_danger 3y ago
      I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
      - kelnos 3y ago
        Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.
        ranger_danger 3y ago
        I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.
      - duskwuff 3y ago
        Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
        ranger_danger 3y ago
        Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.
  - kelnos 3y ago
    I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand. The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8. As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that. [0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map
plorkyeran 3y ago
I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.

You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)

ranger_danger 3y ago
Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
- duskwuff 3y ago
  Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
  - ranger_danger 3y ago
    I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
    - kelnos 3y ago
      Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.
      - ranger_danger 3y ago
        I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.
    - duskwuff 3y ago
      Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
      - ranger_danger 3y ago
        Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.
- kelnos 3y ago
  I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand. The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8. As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that. [0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map

plorkyeran 3y ago

I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.