6 ms·

Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is n

Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.

qwertox 2y ago
You can't assume it if you're handed a random sample of text files.
- duskwuff 2y ago
  You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
  - ranger_danger 2y ago
    Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
    - duskwuff 2y ago
      Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
      - ranger_danger 2y ago
        I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
        kelnos 2y ago
        Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.
        ranger_danger 2y ago
        I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.
        duskwuff 2y ago
        Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
        ranger_danger 2y ago
        Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.
    - kelnos 2y ago
      I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand. The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8. As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that. [0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map
- plorkyeran 2y ago
  I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.
fl7305 2y ago
That comparison doesn't hold. If you're dealing with lengths, you can get input data in meters, centimeters, millimeters, inches, feet, etc. If the input data is human heights, would you automatically assume meters even if the input data is "183"? If the input data is the weight of humans, would you always assume grams, even if the input data is "75"?
- asddubs 2y ago
  would you guess the unit instead of specifying what you expect?
  - fl7305 2y ago
    > would you guess the unit instead of specifying what you expect? It depends on the circumstances. It might be the least bad thing to do. Or not. But that wasn't my point. I replied to this: > I can assume the length of a meter or the weight of a gram Sure, the length of a meter and the "weight" of a gram are both standardized. (To be very picky, "gram" is a mass, not a weight. The actual weight depends on the "g" constant, which on average is 9.81 m/s^2 on earth, but can vary about 0.5%.) So if you know the input is in meters, you don't need to do any further processing. But dealing with input text files with an unknown encoding is like dealing with input lengths with an unknown unit. So while UTF-8 itself might be standardized, it is not the same as all input text files always being in UTF-8. You can choose to say that all input text files must be in valid UTF-8, or the program refuses to load them. Or you can use silent heuristics. Or something inbetween.
    - jujube3 2y ago
      > It depends on the circumstances. It might be the least bad thing to do. Or not. It is not. https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_failure https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
      - fl7305 2y ago
        Is that supposed to be proof that it never depends on the circumstances? Write all software like NASA needs to do for spacecraft? Let me know when anyone has developed a web browser to those standards.
nijave 2y ago
Microsoft would like a word with you (utf-8-bom & utf-16)
- pixl97 2y ago
  Microsoft wants a word with everybody.... and that word is "pay us more money".
paulddraper 2y ago
You can assume encoding is UTF-8, length is in meters, and timezone is UTC. - You just won't always be right.
- kazinator 2y ago
  The trick is to be wrong in precisely those situations in which you don't care.
pyuser583 2y ago
Universal UTF-8 is a hope we aspire to, not a reality we assume.
- fiddlerwoaroof 2y ago
  The way you get that reality is you do the opposite of the recommendation of Postel’s law: be very picky about what you consume and fail loudly if it’s not UTF-8
treflop 2y ago
There’s a difference between asssuming and not making a distinction. Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding. Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.
- jujube3 2y ago
  OK, for a billion dollars I will write a program that deals with KOI-8 or any other text encoding from the 1960s you care to name. Please pay up front.
- Dylan16807 2y ago
  > Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to. Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.

qwertox 2y ago

You can't assume it if you're handed a random sample of text files.

duskwuff 2y ago
You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements. (Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)
- ranger_danger 2y ago
  Shift-JIS specifically cannot be reliably detected, and can pass for UTF-8 but actually make no sense if you looked at a conversion of it.
  - duskwuff 2y ago
    Do you have an example in mind? Looking at the Shift-JIS encoding tables, that seems unlikely to happen in a text of any nontrivial length; there's a small number of Shift-JIS sequences which would be valid as UTF-8, and any meaningfully long text is likely to stray outside that set.
    - ranger_danger 2y ago
      I don't think it's fair to require "meaningfully long text" since when you're dealing with strings in programming they can often be of any arbitrary length.
      - kelnos 2y ago
        Even then, I think it's likely that a your average short byte sequence that's valid Shift-JIS would still not be valid UTF-8.
        ranger_danger 2y ago
        I disagree... half of the UTF-8 Latin-1 supplement range overlaps with the entire half-width katakana alphabet.
      - duskwuff 2y ago
        Encoding detection is usually applied to a larger document, at the point it's ingested into an application. If you're applying it to short strings, something's not right -- where did those strings come from?
        ranger_danger 2y ago
        Taking an ID3 tag example, if you are mass-converting/sanitizing/etc. tag titles and other similar metadata, the strings are often very short, sometimes only even a single codepoint or character, and proper assumptions of encoding can not be relied on because so many people violate specs and put whatever they want in there, which is the whole point of wanting to sanitize the info in the first place.
  - kelnos 2y ago
    I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand. The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8. As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that. [0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map
plorkyeran 2y ago
I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.

fl7305 2y ago

That comparison doesn't hold. If you're dealing with lengths, you can get input data in meters, centimeters, millimeters, inches, feet, etc. If the input data is human heights, would you automatically assume meters even if the input data is "183"? If the input data is the weight of humans, would you always assume grams, even if the input data is "75"?

asddubs 2y ago
would you guess the unit instead of specifying what you expect?
- fl7305 2y ago
  > would you guess the unit instead of specifying what you expect? It depends on the circumstances. It might be the least bad thing to do. Or not. But that wasn't my point. I replied to this: > I can assume the length of a meter or the weight of a gram Sure, the length of a meter and the "weight" of a gram are both standardized. (To be very picky, "gram" is a mass, not a weight. The actual weight depends on the "g" constant, which on average is 9.81 m/s^2 on earth, but can vary about 0.5%.) So if you know the input is in meters, you don't need to do any further processing. But dealing with input text files with an unknown encoding is like dealing with input lengths with an unknown unit. So while UTF-8 itself might be standardized, it is not the same as all input text files always being in UTF-8. You can choose to say that all input text files must be in valid UTF-8, or the program refuses to load them. Or you can use silent heuristics. Or something inbetween.
  - jujube3 2y ago
    > It depends on the circumstances. It might be the least bad thing to do. Or not. It is not. https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_failure https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
    - fl7305 2y ago
      Is that supposed to be proof that it never depends on the circumstances? Write all software like NASA needs to do for spacecraft? Let me know when anyone has developed a web browser to those standards.

nijave 2y ago

Microsoft would like a word with you (utf-8-bom & utf-16)

pixl97 2y ago
Microsoft wants a word with everybody.... and that word is "pay us more money".

paulddraper 2y ago

You can assume encoding is UTF-8, length is in meters, and timezone is UTC. - You just won't always be right.

kazinator 2y ago
The trick is to be wrong in precisely those situations in which you don't care.

pyuser583 2y ago

Universal UTF-8 is a hope we aspire to, not a reality we assume.

fiddlerwoaroof 2y ago
The way you get that reality is you do the opposite of the recommendation of Postel’s law: be very picky about what you consume and fail loudly if it’s not UTF-8

treflop 2y ago

There’s a difference between asssuming and not making a distinction. Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding. Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.

jujube3 2y ago
OK, for a billion dollars I will write a program that deals with KOI-8 or any other text encoding from the 1960s you care to name. Please pay up front.
Dylan16807 2y ago
> Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to. Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.