5 ms·

The comments in this thread are a bit amusing. I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world

The comments in this thread are a bit amusing. I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.

bongodongobob 2y ago
Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.
- fl7305 2y ago
  Even for those people, there is still a ton of old text files in Windows-1252 etc floating around. You can choose to never work on projects where you have to deal with files like that. But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
  - pixl97 2y ago
    Why would you write a tool that does that instead of just digging up one that's already written? This sounds like the folly of writing ones own encryption library.
    - fl7305 2y ago
      Which library do you pick? Which ones are good? Bad? What are the common pitfalls that the library must deal with? How do you know how to evaluate them if all you know is "not UTF-8? Abort with error" ?
      - bongodongobob 2y ago
        As far as implementing new tech goes, this sounds like about the easiest research project you could ask for. Grab a few and test em out. Not sure why you're making this out to be some kind of intractable problem.
        fl7305 2y ago
        If you look for comments by "jcranmer" elsewhere on this page you'll find examples of why it's not as simple as you make it sound. Do you have any experience at all with different charsets under the hood by the way?
        bongodongobob 2y ago
        I do. I cut my IT teeth on an old ass system from the 80s in the 00's. I remember having problems feeding the reports into modern systems. Goofy problems with eol and eof and some other hiccups. It wasn't that bad.
      - pixl97 2y ago
        Honestly if you can't review/read/figure that out without writing a library of your own, you probably shouldn't be writing a library of your own in the first place.
        fl7305 2y ago
        You can't pick and use a library like this without understanding the underlying concepts. That goes for both encryption and the charset conversion issue. It's not always just plug and play. There are examples of where people used encryption libraries in the wrong way and undermined the strength of the encryption (for example, CVE-2024-31497 in PuTTY). A very big part of software development is dealing with leaky abstractions. We don't work with perfect black boxes. We need to understand enough of how things works in the lower layers to avoid problems. Note here that I wrote "enough", not "everything", or "write everything yourself". I would not want a person writing software to handle charset conversion if he refuses to learn how the various encodings work, which charset will decode as another charset or not, etc.
samatman 2y ago
It's more like "I will assume UTF-8 and ignore edge case encoding problems which still arise in Japan, for some strange reason". We are not running short on Unicode codepoints. I'm sure they can spare a few more to cover the Japanese characters and icons which invariably get mentioned any time this subject comes up on HN. I don't know why it hasn't happened and I won't be making it my problem to solve. Best I can do is update to version 16 when it's released.
- koito17 2y ago
  I mention Japanese because I deal with Japanese text daily. I could mention some Chinese documents and sites using GBK to save space (since such encodings use exactly 2 bytes per character whereas the average size in UTF-8 is strictly larger than 2 bytes). But I am not very familiar with it. Overall, I would not say these are "strange reasons".
  - samatman 2y ago
    Other encodings exist, yes. But they can all be mapped to UTF-8 without loss of information[0]. If someone wants to save space, they should use compression, which will reduce any information, regardless of encoding, to approximately the same size. So it's perfectly reasonable to write software on the assumption that data encoded in some other fashion must be first reëncoded as UTF-8 [0]: Except Japanese, people hasten to inform us every time this comes up. Why? Why haven't your odd characters and icons been added to Unicode, when we have cuneiform? That's the strange part. I don't understand why it's the case.
    - Cloudef 2y ago
      Unicode did kind of dumb thing with CJK, the unified Chinese and Japanese kanjis makes displaying the CJK text much harder problem than it should be, as it now relies also on a language specific font to be displayed correctly[0]. I guess this could be bandaided by some sort of language marker in the UTF8 bytestring which then a text shaping engine would have to understand and switch the font accordingly.. 0: https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/
lifthrasiir 2y ago
> ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. Amazingly enough, ARIB STD-B24 is one of the major source of Unicode emojis. So transcoding would actually work for them! (I am aware of some exceptions, but semantically there is no loss.) Unicode and UTF-8 are truly eating every other legacy encoding else, so much so that it is becoming more reasonable to have a separate transcoding step.
- msla 2y ago
  But remember, adding emoji to a character encoding standard is Morally Bad, somehow, and Proof Of Intellectual Decay In The Modern World. Also, Unicode invented emoji to Sap And Impurify Our Bodily Fluids and there is no reason for any of them to exist. /s

bongodongobob 2y ago

Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.

fl7305 2y ago
Even for those people, there is still a ton of old text files in Windows-1252 etc floating around. You can choose to never work on projects where you have to deal with files like that. But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.
- pixl97 2y ago
  Why would you write a tool that does that instead of just digging up one that's already written? This sounds like the folly of writing ones own encryption library.
  - fl7305 2y ago
    Which library do you pick? Which ones are good? Bad? What are the common pitfalls that the library must deal with? How do you know how to evaluate them if all you know is "not UTF-8? Abort with error" ?
    - bongodongobob 2y ago
      As far as implementing new tech goes, this sounds like about the easiest research project you could ask for. Grab a few and test em out. Not sure why you're making this out to be some kind of intractable problem.
      - fl7305 2y ago
        If you look for comments by "jcranmer" elsewhere on this page you'll find examples of why it's not as simple as you make it sound. Do you have any experience at all with different charsets under the hood by the way?
        bongodongobob 2y ago
        I do. I cut my IT teeth on an old ass system from the 80s in the 00's. I remember having problems feeding the reports into modern systems. Goofy problems with eol and eof and some other hiccups. It wasn't that bad.
    - pixl97 2y ago
      Honestly if you can't review/read/figure that out without writing a library of your own, you probably shouldn't be writing a library of your own in the first place.
      - fl7305 2y ago
        You can't pick and use a library like this without understanding the underlying concepts. That goes for both encryption and the charset conversion issue. It's not always just plug and play. There are examples of where people used encryption libraries in the wrong way and undermined the strength of the encryption (for example, CVE-2024-31497 in PuTTY). A very big part of software development is dealing with leaky abstractions. We don't work with perfect black boxes. We need to understand enough of how things works in the lower layers to avoid problems. Note here that I wrote "enough", not "everything", or "write everything yourself". I would not want a person writing software to handle charset conversion if he refuses to learn how the various encodings work, which charset will decode as another charset or not, etc.

samatman 2y ago

It's more like "I will assume UTF-8 and ignore edge case encoding problems which still arise in Japan, for some strange reason". We are not running short on Unicode codepoints. I'm sure they can spare a few more to cover the Japanese characters and icons which invariably get mentioned any time this subject comes up on HN. I don't know why it hasn't happened and I won't be making it my problem to solve. Best I can do is update to version 16 when it's released.

koito17 2y ago
I mention Japanese because I deal with Japanese text daily. I could mention some Chinese documents and sites using GBK to save space (since such encodings use exactly 2 bytes per character whereas the average size in UTF-8 is strictly larger than 2 bytes). But I am not very familiar with it. Overall, I would not say these are "strange reasons".
- samatman 2y ago
  Other encodings exist, yes. But they can all be mapped to UTF-8 without loss of information[0]. If someone wants to save space, they should use compression, which will reduce any information, regardless of encoding, to approximately the same size. So it's perfectly reasonable to write software on the assumption that data encoded in some other fashion must be first reëncoded as UTF-8 [0]: Except Japanese, people hasten to inform us every time this comes up. Why? Why haven't your odd characters and icons been added to Unicode, when we have cuneiform? That's the strange part. I don't understand why it's the case.
  - Cloudef 2y ago
    Unicode did kind of dumb thing with CJK, the unified Chinese and Japanese kanjis makes displaying the CJK text much harder problem than it should be, as it now relies also on a language specific font to be displayed correctly[0]. I guess this could be bandaided by some sort of language marker in the UTF8 bytestring which then a text shaping engine would have to understand and switch the font accordingly.. 0: https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/

lifthrasiir 2y ago

> ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. Amazingly enough, ARIB STD-B24 is one of the major source of Unicode emojis. So transcoding would actually work for them! (I am aware of some exceptions, but semantically there is no loss.) Unicode and UTF-8 are truly eating every other legacy encoding else, so much so that it is becoming more reasonable to have a separate transcoding step.

msla 2y ago
But remember, adding emoji to a character encoding standard is Morally Bad, somehow, and Proof Of Intellectual Decay In The Modern World. Also, Unicode invented emoji to Sap And Impurify Our Bodily Fluids and there is no reason for any of them to exist. /s