4 ms·

Am I reading correctly that characters are 16 bits? If so, is this just for the boundaries (e.g. file system, web)?

TacticalCoder 3y ago
Java predates Unicode 3.1 (I think I got the version correct: but basically when Java was created, Unicode had less than 65536 codepoints). So Java had 16 bits chars from the get go and to this day is still backward compatible with earlier Java code. You can also ask to get the "codepoint", which returns you an int. But, yup, the Java char primitive is 16 bits. It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
- Quekid5 3y ago
  > But to be honest Unicode in itself is probably the biggest SNAFU of our entire field. Explain. EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
  - EnergyAmy 3y ago
    It was always going to be messy to create something that covered all of the writing systems in the world, but things like this didn't help, and is probably what they're referring to: https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
  - tialaramex 3y ago
    I can't explain what the previous poster was thinking, but Unicode is frustrating to work with. The problem is, that frustration just replicates the reality of the human writing systems Unicode captures. Nobody invented weird nonsense to make your life harder in Unicode, millions of humans developed a variety of weird nonsense to make their lives easier, over thousands of years and Unicode captures most of it. Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
    - overboard2 3y ago
      Not every human writing system has to be expressible in all contexts.
      - saagarjha 3y ago
        Popular languages on computers should generally work, though.
    - morelisp 3y ago
      > Nobody invented weird nonsense to make your life harder in Unicode The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)
      - kaba0 3y ago
        I don’t think that pictograms are weird nonsense. Symbology always had an important part in human history, and where else would you put it?
        morelisp 3y ago
        A markup language, or named entities, some length-prefixed blob, just generally something that doesn't require making UAX 29 more difficult. I understand why emoji is as complicated as it is. I don't think it was a good idea to bake that complication into Unicode.
        kaba0 3y ago
        Emoji is not an additional complexity, it builds on top of the same primitives like Chinese characters and whatnot. Unless you claim supporting non-european languages is unnecessary, then emojis are a perfectly good “making use of” of unused regions.
        3y ago
  - cryptonector 3y ago
    Exactly, there's nothing really wrong with Unicode -- it's complicated, yes, but only because human scripts are.
- iforgotpassword 3y ago
  It's the same with the windows API
- cryptonector 3y ago
  Unicode is not a SNAFU. Though the UC has made mistakes here and there, the complexity of Unicode is only a mirror of the complexity of human scripts. Please stop taking dumps on Unicode. It does nothing for actually getting it implemented and implemented well.
  - meepmorp 3y ago
    Seriously, the zoo of text encodings that reigned before widespread Unicode uptake was a pain the ass once you had data that went beyond 7 bits, especially for non-Roman scripts.
  - josefx 3y ago
    > the complexity of Unicode is only a mirror of the complexity of human scripts. Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.
    - deleted 3y ago
      [deleted]
    - cryptonector 3y ago
      > Unicode didn't start out as an encoding covering human scripts > It started out as an encoding to end all already existing encodings And those other encodings were encodings for a variety of human scripts. Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).
      - deleted 3y ago
        [deleted]
za3faran 3y ago
In addition to what the other reply posted, Java has had compact strings for a while now[1], which uses 8-bit `byte` arrays for `String`s, and comes in handy for ISO-8859-1/Latin-1 strings to use a single byte per character. [1] https://openjdk.org/jeps/254 https://openjdk.org/jeps/254
- cesarb 3y ago
  One annoyance with that approach is that a single non-latin-1 codepoint is enough to double the size of the String; if it were an UTF-8 String, it would only add a couple of extra bytes.
  - esprehn 3y ago
    At the expense of O(n) index access. At least Java and JS have compact strings. C# is always UTF-16 because they exposed too much implementation detail to be able to transparently do the compact string optimization.
    - deleted 3y ago
      [deleted]
    - morelisp 3y ago
      > At the expense of O(n) index access. Ah yes, that feature which means we must have 2x memory overhead and/or dynamic dispatch, yet is also totally useless in practice.
      - PhilipRoman 3y ago
        The fact of the matter is that there are millions of places with loops like for(...) {charAt(i)} so it is by definition useful in practice. If Java had pushed iterators more strongly, perhaps it would be possible to make the switch.
        morelisp 3y ago

Java predates Unicode 3.1 (I think I got the version correct: but basically when Java was created, Unicode had less than 65536 codepoints). So Java had 16 bits chars from the get go and to this day is still backward compatible with earlier Java code. You can also ask to get the "codepoint", which returns you an int. But, yup, the Java char primitive is 16 bits. It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.

Quekid5 3y ago
> But to be honest Unicode in itself is probably the biggest SNAFU of our entire field. Explain. EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
- EnergyAmy 3y ago
  It was always going to be messy to create something that covered all of the writing systems in the world, but things like this didn't help, and is probably what they're referring to: https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
- tialaramex 3y ago
  I can't explain what the previous poster was thinking, but Unicode is frustrating to work with. The problem is, that frustration just replicates the reality of the human writing systems Unicode captures. Nobody invented weird nonsense to make your life harder in Unicode, millions of humans developed a variety of weird nonsense to make their lives easier, over thousands of years and Unicode captures most of it. Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
  - overboard2 3y ago
    Not every human writing system has to be expressible in all contexts.
    - saagarjha 3y ago
      Popular languages on computers should generally work, though.
  - morelisp 3y ago
    > Nobody invented weird nonsense to make your life harder in Unicode The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)
    - kaba0 3y ago
      I don’t think that pictograms are weird nonsense. Symbology always had an important part in human history, and where else would you put it?
      - morelisp 3y ago
        A markup language, or named entities, some length-prefixed blob, just generally something that doesn't require making UAX 29 more difficult. I understand why emoji is as complicated as it is. I don't think it was a good idea to bake that complication into Unicode.
        kaba0 3y ago
        Emoji is not an additional complexity, it builds on top of the same primitives like Chinese characters and whatnot. Unless you claim supporting non-european languages is unnecessary, then emojis are a perfectly good “making use of” of unused regions.
        3y ago
- cryptonector 3y ago
  Exactly, there's nothing really wrong with Unicode -- it's complicated, yes, but only because human scripts are.
iforgotpassword 3y ago
It's the same with the windows API
cryptonector 3y ago
Unicode is not a SNAFU. Though the UC has made mistakes here and there, the complexity of Unicode is only a mirror of the complexity of human scripts. Please stop taking dumps on Unicode. It does nothing for actually getting it implemented and implemented well.
- meepmorp 3y ago
  Seriously, the zoo of text encodings that reigned before widespread Unicode uptake was a pain the ass once you had data that went beyond 7 bits, especially for non-Roman scripts.
- josefx 3y ago
  > the complexity of Unicode is only a mirror of the complexity of human scripts. Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.
  - deleted 3y ago
    [deleted]
  - cryptonector 3y ago
    > Unicode didn't start out as an encoding covering human scripts > It started out as an encoding to end all already existing encodings And those other encodings were encodings for a variety of human scripts. Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).
    - deleted 3y ago
      [deleted]

za3faran 3y ago

In addition to what the other reply posted, Java has had compact strings for a while now[1], which uses 8-bit `byte` arrays for `String`s, and comes in handy for ISO-8859-1/Latin-1 strings to use a single byte per character. [1] https://openjdk.org/jeps/254 https://openjdk.org/jeps/254

cesarb 3y ago
One annoyance with that approach is that a single non-latin-1 codepoint is enough to double the size of the String; if it were an UTF-8 String, it would only add a couple of extra bytes.
- esprehn 3y ago
  At the expense of O(n) index access. At least Java and JS have compact strings. C# is always UTF-16 because they exposed too much implementation detail to be able to transparently do the compact string optimization.
  - deleted 3y ago
    [deleted]
  - morelisp 3y ago
    > At the expense of O(n) index access. Ah yes, that feature which means we must have 2x memory overhead and/or dynamic dispatch, yet is also totally useless in practice.
    - PhilipRoman 3y ago
      The fact of the matter is that there are millions of places with loops like for(...) {charAt(i)} so it is by definition useful in practice. If Java had pushed iterators more strongly, perhaps it would be possible to make the switch.
      - morelisp 3y ago