7 ms·

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

ChrisSD 6y ago
Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.
- BlueTemplar 6y ago
  One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.
  - cygx 6y ago
    Not all user-perceived characters can be represented as a single Unicode codepoint. Hence, Unicode text encodings (almost[1]) always have to be treated as variable length, even UTF-32. [1] at runtime, you could dynamically assign 'virtual' codepoints to grapheme clusters and get a fixed-length encoding for strings that way
    - jart 6y ago
      Even the individual unicode codepoints themselves are variable width if we consider that things like cjk and emoji take up >1 monospace cells.
      - lanstin 6y ago
        Every time I see one of these threads, my gratitude to only do backend grows. Human behavior is too complex, let the webdevs handle UI, and human languages are too complex, not sure what speciality handles that. Give me out of order packets and parsing code that skips a character if the packet length lines up just so any day. I am thankful that almost all the Unicode text I see is rendered properly now, farewell the little boxes. Good job lots of people.
        jart 6y ago
        I think we really have the iPhone jailbreakers to thank for that. U.S. developers were allergic almost offended by anything that wasn't ASCII and then someone released an app that unlocked the emoji icons that Apple had originally intended only for Japan. Emoji is defined in the astral planes so almost nothing at the time was capable of understanding them, yet were so irresistible that developers worldwide who would otherwise have done nothing to address their cultural biases immediately fixed everything overnight to have them. So thanks to cartoons, we now have a more inclusive world.
        londons_explore 6y ago
        I'm pretty sure Unicode was pretty widespread before the iphone/emoji popularity.
        cygx 6y ago
        There's supporting Unicode, and 'supporting' Unicode. If you're only dealing with western languages, it's easy to fall into the trap of only 'supporting' Unicode. Proper emoji handling will put things like grapheme clusters and zero-width joiners on your map.
  - kortex 6y ago
    > One byte = one "character" makes for much easier programming. Only if you are naively operating in the Anglosphere / world where the most complex thing you have to handle is larger character sets. In reality, there's ligatures, diacritics, combining characters, RTL, nbsp, locales, and emoji (with skin tones!). Not to mention legacy encoding. And no, it does not use a "small fraction of memory and storage" in a huge range of applications, to the point where some regions have transcoding proxies still.
    - BlueTemplar 6y ago
      This is not about covering ALL of Unicode. This is about starting to cover Unicode. "Anglosphere" would be just 7(&"8") bit ASCII, and it's the current situation where it takes quite a lot of skill and knowledge just to start learning how to properly deal with Unicode, because it's often not even taught ! IMHO 32-bit bytes would help tremendously with onboarding developers into Unicode, because it would force dumping ASCII-only as the starting point (and sadly, often ending point) for teaching how to deal with text. And who can blame the teachers, Unicode is already hard enough without even having to deal with the difficulties coming from having to explain its multi-byte representation... Last but not least : this would have forced standardization between the Unix world now on UTF-8 and the Windows world which is still stuck on UTF-16 (and Windows-1252 ?!?) for some of the core functions like filenames, which, for instance, still regularly results in issues working with files with non-ASCII filenames.
- a1369209993 6y ago
  > Unicode is a variable width abstract encoding; To be a bit more explicit: Unicode is a character encoding, to 20-and-a-half-bit 'bytes', that is variable-width in those 'bytes', even before considering how the 'bytes' are encoded to actual bytes. Eg "ψ̊" (greek small psi with ring above) is U+3C8 U+30A (two 'bytes').
- spacechild1 6y ago
  Unicode is not a text encoding. UTF8, UTF16, UTF32, etc. are text encodings. > a single character can be made up of multiple code points. It's really the other way round...
  - spacechild1 6y ago
    Or maybe you meant to say: A single abstract character (or code point) can be made up of multiple code units. Unfortunately, the term “character“ alone is ambiguous because depending on the context it can refer to either code points or code units.
    - ChrisSD 6y ago
      To be technical, by "character" I mean "user-perceived character" or (in Unicode speak) "extended grapheme cluster". This is the thing a user will think of as one character when looking at it on their screen. A code point is the atomic unit of the abstract Unicode encoding. By "abstract" I mean it is not an actual text encoding you can write to a file. A code unit is the atomic unit of an actual text encoding, such as UTF-8, UTF-16LE or UTF-32LE (and their BE equivalents). --- So to put it together a "user-perceived character" is made up of one or more "code points". When implemented in an application, each "code point" is encoded using one or more "code units".
      - spacechild1 6y ago
        Now that's a good summary! When talking about Unicode, some extra clarity can never hurt.
wongarsu 6y ago
People were holding off on transitioning because pointers use twice as much space in x64. If bytes had quadrupled in space with x64 we would still be using 32 bit software everywhere
- BlueTemplar 6y ago
  Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory. And do you have examples of still widely used 8-bit sized data formats ?
  - jart 6y ago
    RGB and Y′CbCr
    - BlueTemplar 6y ago
      To start with, RGB (and I assume Y′CbCr ?) can be encoded in many different ways. The most common one today (still) uses 8 bits per channel, meaning that a separated 1-octet value can only define monochrome. Therefore 8-bpc RGB is a 24-bit sized format, not a 8-bit sized data one. And, by an interesting coincidence, with the arrival of "HDR", 8-bit per channel is slowly becoming obsolete (because insufficient). The next "step" is 10-bit per channel with 3 channels (hence "HDR10(+)"), and so should fit quite well in 32 bits ? (However, it would seem that even Dolby's Perceptual Quantizer transfer function might need 12 bits per channel to avoid banding over the "HDR" Rec.2020/2100-sized color gamut..?)
      - jart 6y ago
        We're debating semantics, but if I reshaped an RGB image into component arrays i.e. u8[yn][xn][3] → u8[3][yn][xn] then would you still view that as a 24-bit format? What if those 24-bit values were huffman or run-length encoded would it be an n-bit format? If your Y′CbCr luminance plane has a legal range of 16..235 and the chrominance planes are 16..240, then would it be a 23.40892 bit format?
jart 6y ago
Use Erlang. It has 32-bit char.
- toast0 6y ago
  Not really. Strings are a list of integers [1], integers are signed and fill a system word, but there's also 4 bits of type information. So you can have a 28-bit signed integer char on a 32-bit system or a signed 60-bit integer. However, since Unicode is limited to 21-bits by utf-16 encoding, a unicode code point will fit in a small integer. [1] unless you use binaries, which is often a better choice.
spacechild1 6y ago
You know, bytes are not only about text, they are also used to represent binary data... Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.
- BlueTemplar 6y ago
  https://news.ycombinator.com/item?id=27086928 https://news.ycombinator.com/item?id=27086928 These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !
  - spacechild1 6y ago
    But Unicode itself doesn't! Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!
    - BlueTemplar 6y ago
      My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again, as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)
      - spacechild1 6y ago
        But why? Just so that we need to do more bit twiddling and waste memory? Again, bytes are not foremost about text. We habe to deal with all sorts of data, many of which is shorter than 32 bits. You can always pick a larger data type for your type of work, but not the opposite.
        BlueTemplar 6y ago
        Because these days it's critical for "basic computer literacy" : https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27104860 https://news.ycombinator.com/item?id=27104860 (You'll also notice that caring about not wasting the 8th bit with ASCII has lead us into all sorts of issues... and why care so much about it when as soon as data density becomes important, we can use compression which AFAIK easily rids us of padding ?)

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

BlueTemplar 6y ago
One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.
- cygx 6y ago
  Not all user-perceived characters can be represented as a single Unicode codepoint. Hence, Unicode text encodings (almost[1]) always have to be treated as variable length, even UTF-32. [1] at runtime, you could dynamically assign 'virtual' codepoints to grapheme clusters and get a fixed-length encoding for strings that way
  - jart 6y ago
    Even the individual unicode codepoints themselves are variable width if we consider that things like cjk and emoji take up >1 monospace cells.
    - lanstin 6y ago
      Every time I see one of these threads, my gratitude to only do backend grows. Human behavior is too complex, let the webdevs handle UI, and human languages are too complex, not sure what speciality handles that. Give me out of order packets and parsing code that skips a character if the packet length lines up just so any day. I am thankful that almost all the Unicode text I see is rendered properly now, farewell the little boxes. Good job lots of people.
      - jart 6y ago
        I think we really have the iPhone jailbreakers to thank for that. U.S. developers were allergic almost offended by anything that wasn't ASCII and then someone released an app that unlocked the emoji icons that Apple had originally intended only for Japan. Emoji is defined in the astral planes so almost nothing at the time was capable of understanding them, yet were so irresistible that developers worldwide who would otherwise have done nothing to address their cultural biases immediately fixed everything overnight to have them. So thanks to cartoons, we now have a more inclusive world.
        londons_explore 6y ago
        I'm pretty sure Unicode was pretty widespread before the iphone/emoji popularity.
        cygx 6y ago
        There's supporting Unicode, and 'supporting' Unicode. If you're only dealing with western languages, it's easy to fall into the trap of only 'supporting' Unicode. Proper emoji handling will put things like grapheme clusters and zero-width joiners on your map.
- kortex 6y ago
  > One byte = one "character" makes for much easier programming. Only if you are naively operating in the Anglosphere / world where the most complex thing you have to handle is larger character sets. In reality, there's ligatures, diacritics, combining characters, RTL, nbsp, locales, and emoji (with skin tones!). Not to mention legacy encoding. And no, it does not use a "small fraction of memory and storage" in a huge range of applications, to the point where some regions have transcoding proxies still.
  - BlueTemplar 6y ago
    This is not about covering ALL of Unicode. This is about starting to cover Unicode. "Anglosphere" would be just 7(&"8") bit ASCII, and it's the current situation where it takes quite a lot of skill and knowledge just to start learning how to properly deal with Unicode, because it's often not even taught ! IMHO 32-bit bytes would help tremendously with onboarding developers into Unicode, because it would force dumping ASCII-only as the starting point (and sadly, often ending point) for teaching how to deal with text. And who can blame the teachers, Unicode is already hard enough without even having to deal with the difficulties coming from having to explain its multi-byte representation... Last but not least : this would have forced standardization between the Unix world now on UTF-8 and the Windows world which is still stuck on UTF-16 (and Windows-1252 ?!?) for some of the core functions like filenames, which, for instance, still regularly results in issues working with files with non-ASCII filenames.
a1369209993 6y ago
> Unicode is a variable width abstract encoding; To be a bit more explicit: Unicode is a character encoding, to 20-and-a-half-bit 'bytes', that is variable-width in those 'bytes', even before considering how the 'bytes' are encoded to actual bytes. Eg "ψ̊" (greek small psi with ring above) is U+3C8 U+30A (two 'bytes').
spacechild1 6y ago
Unicode is not a text encoding. UTF8, UTF16, UTF32, etc. are text encodings. > a single character can be made up of multiple code points. It's really the other way round...
- spacechild1 6y ago
  Or maybe you meant to say: A single abstract character (or code point) can be made up of multiple code units. Unfortunately, the term “character“ alone is ambiguous because depending on the context it can refer to either code points or code units.
  - ChrisSD 6y ago
    To be technical, by "character" I mean "user-perceived character" or (in Unicode speak) "extended grapheme cluster". This is the thing a user will think of as one character when looking at it on their screen. A code point is the atomic unit of the abstract Unicode encoding. By "abstract" I mean it is not an actual text encoding you can write to a file. A code unit is the atomic unit of an actual text encoding, such as UTF-8, UTF-16LE or UTF-32LE (and their BE equivalents). --- So to put it together a "user-perceived character" is made up of one or more "code points". When implemented in an application, each "code point" is encoded using one or more "code units".
    - spacechild1 6y ago
      Now that's a good summary! When talking about Unicode, some extra clarity can never hurt.

wongarsu 6y ago

People were holding off on transitioning because pointers use twice as much space in x64. If bytes had quadrupled in space with x64 we would still be using 32 bit software everywhere

BlueTemplar 6y ago
Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory. And do you have examples of still widely used 8-bit sized data formats ?
- jart 6y ago
  RGB and Y′CbCr
  - BlueTemplar 6y ago
    To start with, RGB (and I assume Y′CbCr ?) can be encoded in many different ways. The most common one today (still) uses 8 bits per channel, meaning that a separated 1-octet value can only define monochrome. Therefore 8-bpc RGB is a 24-bit sized format, not a 8-bit sized data one. And, by an interesting coincidence, with the arrival of "HDR", 8-bit per channel is slowly becoming obsolete (because insufficient). The next "step" is 10-bit per channel with 3 channels (hence "HDR10(+)"), and so should fit quite well in 32 bits ? (However, it would seem that even Dolby's Perceptual Quantizer transfer function might need 12 bits per channel to avoid banding over the "HDR" Rec.2020/2100-sized color gamut..?)
    - jart 6y ago
      We're debating semantics, but if I reshaped an RGB image into component arrays i.e. u8[yn][xn][3] → u8[3][yn][xn] then would you still view that as a 24-bit format? What if those 24-bit values were huffman or run-length encoded would it be an n-bit format? If your Y′CbCr luminance plane has a legal range of 16..235 and the chrominance planes are 16..240, then would it be a 23.40892 bit format?

jart 6y ago

Use Erlang. It has 32-bit char.

toast0 6y ago
Not really. Strings are a list of integers [1], integers are signed and fill a system word, but there's also 4 bits of type information. So you can have a 28-bit signed integer char on a 32-bit system or a signed 60-bit integer. However, since Unicode is limited to 21-bits by utf-16 encoding, a unicode code point will fit in a small integer. [1] unless you use binaries, which is often a better choice.

spacechild1 6y ago

You know, bytes are not only about text, they are also used to represent binary data... Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.

BlueTemplar 6y ago
https://news.ycombinator.com/item?id=27086928 https://news.ycombinator.com/item?id=27086928 These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !
- spacechild1 6y ago
  But Unicode itself doesn't! Anyway, it doesn't make much sense to define the size of a “byte“ as anything else then 8 bits, because that's the smallest adressable memory unit. If you need a 32 bit data type, just use one!
  - BlueTemplar 6y ago
    My very point is that we should have increased the size of the smallest addressable memory unit from 8 to 32 bits, increased again, as previous computer architectures used from 4 to 7 bits per byte. (There might be still e-mail servers around directly compatible with "non-padded" 7-bit ASCII ?)
    - spacechild1 6y ago
      But why? Just so that we need to do more bit twiddling and waste memory? Again, bytes are not foremost about text. We habe to deal with all sorts of data, many of which is shorter than 32 bits. You can always pick a larger data type for your type of work, but not the opposite.
      - BlueTemplar 6y ago
        Because these days it's critical for "basic computer literacy" : https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27094663 https://news.ycombinator.com/item?id=27104860 https://news.ycombinator.com/item?id=27104860 (You'll also notice that caring about not wasting the 8th bit with ASCII has lead us into all sorts of issues... and why care so much about it when as soon as data density becomes important, we can use compression which AFAIK easily rids us of padding ?)