2 ms·
The industry moved more than two decades ago already. However operating systems continue to do the wrong thing for no good reason other than that they have been
by jillesvangurp 4y ago
The industry moved more than two decades ago already. However operating systems continue to do the wrong thing for no good reason other than that they have been doing that since forever. UTF-8 was already a thing when Java launched in 1995. It was proposed as a standard three decades ago. As I mentioned, I've been dealing with weird UTF-8 related issues throughout my career. As soon if you have APIs that rely on the default and operating systems that don't set that to UTF-8, you end up with corrupted content.
There are all sorts of ways to make this happen: APIs that set a mime type but no character encoding, progams that open and save files, databases that don't default to using utf-8 (or like mysql actually have some non standard 3 byte version of utf-8 that doesn't deal with emojis and a lot of other characters). Linux distributions don't set UTF-8 as the character encoding until late in the boot process. IMHO there's no good reason why it should ever do anything else than UTF-8 in a shell from the microsecond it boots until it shuts down. Changing all that is hard however. Not for technical reasons but for reasons of backwards compatibility with systems that are arguably part of the problem here.
I think the root cause for this might be a blind spot that people in English speaking countries have since they don't use a lot of special characters and therefore don't see the problem. As soon as you start using Spanish, Swedish, German, etc. You are in trouble. And forget about any non latin scripts. Latin-1 is not appropriate for any of those. Utf-8 is a better default that should be usable everywhere. Some countries might prefer utf-16. But that's mainly a memory optimization that should not matter much in 2023.