5 ms·

For older programmers like me, who pretty much only deal in English, the very simplicity of ASCII can make unicode difficult to grasp. In ASCII 1 byte (always

For older programmers like me, who pretty much only deal in English, the very simplicity of ASCII can make unicode difficult to grasp. In ASCII 1 byte (always 1 byte) of ram = 1 character. And the encoding called ASCII matches the mapping called ASCII. Along comes Unicode (a mapping), which has multiple different encodings. A difficult distinction leading to statements like "that's a unicode string/file/field". Along comes unicode which has variable bytes per character. (Yes, even for utf-32, which is why no-one uses utf-32). I still regularly come across folk who think unicode means 2 bytes per character. Along comes unicode which asks you to consider if the functions LEN, SUB, SLICE etc are counting Bytes or Characters. Along comes unicode that breaks the idea of setting field length in databases as "number of characters", made worse by a cohort brought up on limited storage, who want to be "effecient" in their declarations. Along comes unicode with code units, code points, characters -all of which come into play, all with different, or variable, lengths. So while utf-21 might be a "toy", even making something like that yourself is a learning exercise well worth the endeavour. It should be a mandatory teaching exercise. Two thumbs up.

franciscop 3y ago
As a beginner programmer back in the day I'd have agreed that unicode is weird and "length in characters" is the right metric for the database entry. However as a senior dev, who have read and messed around enough with binary, UTF-X, compression/GZIP, etc, I'd say that "character length" for database field size is a weird concept and that "size in bytes" would make more sense since that maps better to what you have in the HDD/SSD/network.
- hans_castorp 3y ago
  > I'd say that "character length" for database field size is a weird concept Until you want a constraint for such a column. From a business perspective, you would limit the column to a specific number of characters, not bytes
  - bruce511 3y ago
    Let me be clear. You currently cannot constrain for number of characters in a unicode string. From a business perspective it's unhelpful to think of constraining in that way. By all means constrain on bytes, and if that's close enough for you, use that as a proxy. Or provision lots of space, and count characters in the UI.
  - qalmakka 3y ago
    This is ALWAYS broken on unicode. It only makes sense in ASCII and with alphabets similar to the Latin alphabet.
    - Dylan16807 3y ago
      It's not broken if "character" means "extended grapheme cluster". Though in that case it makes it very hard to index the column.
      - qalmakka 3y ago
        Nobody cares about them, I am afraid. Most people _really_ can't accept that you can't work on Unicode like you did with ASCII, and do not want to let go of their old C habits (like iterating char by char and doing something).
  - PeterisP 3y ago
    If you want a constraint for such a column, then due to nature of our complex writing, it has to be separate from the actual field type and physical storage allocation - it's the equivalent of an integer field having a constraint that it must be between 1 and 100.
  - twic 3y ago
    Is there ever a genuine, non-arbitrary business requirement to limit any string to a certain number of unicode characters? And by characters here we probably mean extended grapheme clusters. If the data is going to get printed in a monospace font, on a passport or credit card or something, then i can understand a limit on the number of characters. But then it's not full unicode either - you want to constrain the length in some specific character set. Otherwise, i think limits are always arbitrary. I suspect there is a strong cultural holdover from the days of punched cards here.
- HelloNurse 3y ago
  Typically there should be a maximum size in bytes for performance reasons (e.g. how many rows fit in a page) and/or a minimum size in "characters" (very ill defined but usually approximated as code points, with some slack to allow a reasonable amount of combining marks in addition to the desired number of letters), with no guarantee that the two sizes are compatible. I've often found in practice short UTF-8 columns with a bytes length, that "accidentally" truncated text with ctastrophic effects after the application checked value lengths in characters.
- layer8 3y ago
  It makes it less intuitive for the user if you have input fields limited to n bytes instead of to n characters (complications caused by combining characters notwithstanding).
  - colejohnson66 3y ago
    Twitter used(?) to have this problem: emojis, not being in the BMP, would count as two "characters" because they were encoded as UTF-16 surrogate pairs.
- franciscop 3y ago
  Everyone replying here seems to be quoting "business reasons" which seem very English-centric. Try to talk to a Japanese or Korean person about their "business reasons". I am suffering the opposite problem. My name has 36+ letters and I live in Japan, where the average full name has "3-4 characters" (Kanji) so they tend to be safe and allow for 10-15 characters, if I'm lucky maybe even 10 for given and 10 for family names (20 in total), where I don't have to totally murder my name and just need to amputate it.
bawolff 3y ago
> I still regularly come across folk who think unicode means 2 bytes per character. To be fair, a lot of programming interfaces with "unicode" in the name were invented when UCS-2 was the only option.
- bruce511 3y ago
  True. And for those of us around who "learned" unicode in the 90s, when unicode was UCS2, it can be hard to let go of those "truths that are no longer true." This complicates the learning process now.
  - pastage 3y ago
    From Wikipedia. > In 1996, a surrogate character mechanism was implemented in Unicode 2.0, so that Unicode was no longer restricted to 16 bits. This increased the Unicode codespace to over a million code points, So that seems like a fair analysis. I know we still did latin-1 DBs here into 2005, but I think you had to choose charsets at that point. Everyone knew you should do Unicode but there were so many excusese not to. Unicode still is harder than ascii because of that mindset.
rcme 3y ago
None of your complaints make any sense. If you limit your usage to ASCII characters, then all of your complaints of Unicode no longer apply. E.g. character length matches your intuition of “number of characters”.
- bruce511 3y ago
  Firstly we live in a world where limiting yourself to ASCII characters is increasingly untenable. There are plenty of non-ascii characters in Spanish or European names. So your argument about limiting to ASCII is like saying its also not a problem if we don't use computers. Sure, you're right, but it's not terribly practical. Plus your argument uses the word Unicode (a mapping) where it only applies to utf-8, (a specific encoding ). This inaccurate use of the terms is the very issue I'm alluding to. Incidentally I'm not complaining about Unicode. Unicode is necessary and good. My comment is that (mostly older, but not exclusively) programmers find it hard to grasp the nuances of unicode and think of it as a encoding, not a mapping.
  - rcme 3y ago
    > Firstly we live in a world where limiting yourself to ASCII characters is increasingly untenable. I agree. But your original comment bemoaned the difference between using ASCII and Unicode encodings. I was just pointing out that any Unicode encoding is not variable length of you limit yourself to ASCII. This applies to all Unicode encodings, not just utf-8.
    - bruce511 3y ago
      I might not have been clear in the original posting. I wasn't intending to bemoan the difference, I was bemoaning old programmers (like me) not being familiar with the most basic unicode concepts, which leads to lots of issues. Yes, if you limit your strings to ASCII then they are always 1 byte long (utf-8), or 2 bytes long (utf-16) or 4 bytes long (utf-32). So you still need to know which encoding is in play. But you can then randomly access "the 4th character". My follow up points out though that this is meaningless. Because in the real world you cannot limit your strings to just ASCII in the first place. So your point is moot from that perspective.
jesprenj 3y ago
That's why in C you have wchar_t. https://en.wikipedia.org/wiki/Wide_character https://en.wikipedia.org/wiki/Wide_character
- qalmakka 3y ago
  wchar_t is a massive mistake that came from an era (the '90s) where people wanted to delude themselves they could keep ASCII-like code ergonomics (iterating, random access, ...) by just changing character types. It was a stupid mistake we are still paying to this day, when OSes and libraries made in the '90s are forced to convert from the UCS2 they use internally to UTF-8 constantly (just look at Qt). The cherry on top is also that `wchar_t` is spectacularly broken in C, alongside multichar encoding and everything that's not ASCII (https://thephd.dev/cuneicode-and-the-future-of-text-in-c https://thephd.dev/cuneicode-and-the-future-of-text-in-c).
  - jesprenj 3y ago
    Isn't this similar to what modern languages did? They abstracted away the underlying encoding so that the programmer deals with characters instead of bytes. Two examples are Python and Javascript strings. They kept ASCII-like code ergonomics (iterating and random access). That of course required that they separated strings and byte-like objects -- like uint8_t and wchar_t. Isn't signed char actually the culprit in modern C and therefore useless? It's main use is ASCII and that's obsolete.
    - qalmakka 3y ago
      > so that the programmer deals with characters instead of bytes This is a broken approach and a broken mindset. Modern languages like Rust (Python is not modern anymore, it is ATM 33 years old which is older than C was in 1990) reverted back to "a string is an array of uint8" because that's the only sane way to operate on them. Naive iteration and random access are broken unless they are performed on the underlying bytes, because iterating on Unicode characters is a *broken concept*. Python strings are also arguably somewhat broken, because they still allow random access into a string at "character" (not byte) indexes which causes all sorts of issues when slicing. This means that code that works perfectly with English text will malfunction when handling other languages. The hard truth is that slicing an Unicode string is a non-trivial and (somewhat) expensive operation, while Python slicing was designed in Python < 2.7 with the assumption char == byte, which now is always broken in every encoding except 8 bit, single codepage ones. For instance: >>> print(s) 'Crêpe' >>> [unicodedata.name(c) for c in s] ['LATIN CAPITAL LETTER C', 'LATIN SMALL LETTER R', 'LATIN SMALL LETTER E', 'COMBINING CIRCUMFLEX ACCENT', 'LATIN SMALL LETTER P', 'LATIN SMALL LETTER E'] >>> len(s) 6 In this case, the string "Crêpe" is in NFD form (all decomposable characters are decomposed; in particular `ê` is not U+00EA 'LATIN SMALL LETTER E WITH CIRCUMFLEX' but '\u0065\u0302', which is U+0065 'LATIN SMALL LETTER E' plus U+0302 'COMBINING CIRCUMFLEX ACCENT' ( ◌̂ ) In Rust, which is more modern, enforces UTF-8 (and not some broken version of UCS-2 or worse) slicing is ALWAYS done on bytes because it doesn't make sense to perform it on "codepoints". Asking for a slice with a byte range that falls in the middle of an UTF-8 multichar sequence will cause a panic; this also means that if you do s[2] you get the second _byte_, not "char". If you want the second character, you are forced to go through a Unicode library, as you always should. Python will instead happily comply, basically returning only the codepoints you asked, because it sees a string composed of 6 Unicode codepoints despite the fact the user only sees 5 rendered on screen: >>> s[0:3] 'Cre' >>> [unicodedata.name(c) for c in s[0:3]] ['LATIN CAPITAL LETTER C', 'LATIN SMALL LETTER R', 'LATIN SMALL LETTER E'] This makes string slicing basically useless, because even if you normalise all strings using `unicodedata.normalize('NFC', s)` before performing slicing on them, there are still several printable characters which are not represented with a single Unicode codepoint. For instance, >>> eu = '' >>> len(eu) 2 >>> x[0:1] '' because all flag emojis are represented with two Unicode codepoints, each one representing a letter. TL;DR: do not use wchar_t, UTF-16, UTF-32, ... only use UTF-8 if possible and under all circumstances treat strings a special versions as black boxes specialised for text you can only access byte per byte. If you need to do text operations, use a library like ICU or whatever your language/repository provides.
the_mitsuhiko 3y ago
Maybe but for people who worked a lot with code pages / different ISO encodings unicode brings simplicity that wasn’t there before.
nayuki 3y ago
In ASCII, the concept of a character is overloaded with many properties. Unicode breaks "character" out to separate concepts like code unit, code point, grapheme cluster, etc. Because of this, some of your statements are invalid. For example, "Along comes unicode which has variable bytes per character. (Yes, even for utf-32, which is why no-one uses utf-32).". UTF-32 has a fixed number of bytes per code point. It doesn't have a fixed number of bytes per grapheme cluster.
- bruce511 3y ago
  Correct. I used character where you used grapheme cluster. Utf-32 is variable length at the character (aka grapheme cluster) level. If there are any other invalid parts please elaborate - I'm always keen to learn (seriously).
qalmakka 3y ago
The gist of unicode is that code like const char *const end = str + strlen(str); for (const char *it = str; it < end; ++i) { .. } is almost always wrong unless your locale is C and you explicitly say you only work with ASCII. It's wrong with UTF-32 and char32_t too, because as the post suggests Unicode codepoints are not glyphs - you can have glyphs spanning over multiple codepoints, with multiple normalization forms. This is particularly tricky, because characters printed the same can be represented in multiple ways and it's hard to discern them, i.e. è can be a single codepoint or a combination of a '◌́ ' modifier and 'e'. In general the trivial C assumption "strings are just char arrays" most people are accustomed to is dangerous and broken in Unicode. str[x] in Unicode is wrong 99% of the time, thus UTF16 and 32 are nonsense¹ and either you treat strings as bytes you just pass around, or you need a full Unicode library such as ICU to operate on Unicode text safely. ¹: in my experience they are NEVER used mostly, UTF-16 is actually two incompatible encodings and use it as a way to "UTF-wash" the broken UCS-2 they so eagerly adopted in the '90s (including Java, Windows NT, Cocoa, Qt, Unreal, JavaScript, [partially] Python, ...) .
- electroly 3y ago
  UTF-16 is the internal encoding of ICU to this day. If you're using ICU, you're using UTF-16. The library treats UTF-8 as a conversion target rather than a native representation. If you ever see a new project pick UTF-16 and you don't know why, it's because of ICU; any other choice forces a round trip conversion on every ICU call. If you pick UTF-16 you can just use icu::UnicodeString as your string representation and life is easy.
- johnnyanmac 3y ago
  So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to iterate before I get a proper glyph instead of part of a glyph, or worse: a glyph and some extra codepoints? Is the concept so esoteric that it's best to just find a blackbox library and not worry about the finer details?
  - HelloNurse 3y ago
aardvark179 3y ago
That ship sailed long before Unicode. ISO/IEC 8859-1 Included several accents which combine with a preceding character, and that was standardised in 1987. If you only deal in English then remove all accents and map everything down to 7 bit ASCII, if you don’t just deal with English then accept that and deal with it. :-) Having had to do some internationalisation of code written by Americans that couldn’t even handle 8 bit characters I am fine with spreading this bit of pain a little more evenly round the world.
- dspillett 3y ago
  > ISO/IEC 8859-1 … standardised in 1987 Are you sure about that designation? I don't remember 8859-1 having combining characters and other sources (https://en.wikipedia.org/wiki/ISO/IEC_8859-1 https://en.wikipedia.org/wiki/ISO/IEC_8859-1 for one) seem to agree and also list it as being officially ratified later than '87 ('98).
unconed 3y ago
UTF-8 is 30 years old, and 20 years ago it was already in common use on the web. I know because I was the doofus who pushed PHP projects to adopt it because monolinguals kept fucking up character encodings. Being an older programmer is a very poor excuse at this point to not understand Unicode.
- bruce511 3y ago
  >> Being an older programmer is a very poor excuse at this point to not understand Unicode. I completely agree. Which does not make the ignorance I experience around Unicode from both older, and younger programmers, easier to forgive.
euroderf 3y ago
> the very simplicity of ASCII can make unicode difficult to grasp. Somewhat related: I hate "smart" "quotes". I find myself editing them into plain old << " >> and << ' >>. If you want to use "smart" quotes to add semantic meaning to text - as in "this is a quote" - then you should be using markup to do the job. Stop abusing non-ASCII typography so you can pretend to write "semantic" material that you think will impress all the cool kids that know structured documentation. Who else remembers that great Perl script, "The Demoronizer" ?
- chrismorgan 3y ago
  I too hate “smart quotes”, meaning anything that automatically converts from ASCII, but only for myself. That’s because I’m a weirdo that types exactly what I mean, things like em dashes and en dashes and curly quotes and such, using my Compose key. Opening and closing quotation marks are different from one another (simple fact, true from the very early days of quotation marks), so they should be represented differently in Unicode (simple fact). Anything along the lines of “smart quotes” is just a pragmatic approach to let normal people get the correct quotation marks, and I’m mostly glad they can have it (I just wish it’d handle things like “’cos” and “’90s” properly, which use an apostrophe which is equivalent to a right single quotation mark; but other than that, I’m glad when people use curly quotes rather than straight quotes), so long as I don’t have to have it (and I’m glad to say I don’t).
JdeBP 3y ago
Older programmers like you are not old enough. (-: ASCII had variable numbers of bytes per glyph back in the 1960s, and the Teletype Model 37 semantics of combining-backspace and combining-underscore still exist in modern operating systems, like the various BSDs and Linux-based ones, to this day. It's a combining-characters multibyte encoding understood by programs from less(1) through ul(1) to nroff(1). ASCII had multiple transfer encodings, from 8N1 to 7E2, many of which were neither 7-bit nor 8-bit; and 1 byte of RAM was not 1 ASCII character. 1 byte of RAM not being 1 ASCII character is, after all, what brought about "extended ASCII". Along with that came ECMA-35, with its GR and GL, and code pages. All of those broke theretofore held truths such as that one could just strip the parity bit, actually introduced the false idea (that many people had to keep correcting) that ASCII was 8-bit, and required a full-blown state machine for parsing ECMA-35 escape sequences. Then there are the tales that old programmers can tell about how they couldn't actually deal in English because ASCII didn't have a pound symbol, and what was "English" was actually American (which was, after all, the "A" in "ASCII"). There is a whole generation of older programmers who laugh hollowly at the idea of "the simplicity of ASCII"; when the amount of time that they spent fiddling with serial protocol settings, transcoding between code pages, handling language variant characters in Teletext, wondering why an 8-bit-clean transport protocol such as TCP didn't mean that SMTP could be 8-bit clean, and doing other such stuff, probably came to years in total spent on all of this "simplicity".
- msla 3y ago
  > Then there are the tales that old programmers can tell about how they couldn't actually deal in English because ASCII didn't have a pound symbol What's #, then? Chopped liver? /s
  - pwdisswordfishc 3y ago
    The number sign, of course.
IIAOPSW 3y ago
You don't go far enough. Lets do some language reform. The entire alphabet is always just 5 bits expressed in different mediums. Lets make the written characters are all based on 5 sticks coming out of a central point literally representing your hand. Sign language, pretty obvious. Morse code, the up / down fingers directly translate to holding the key down or not. Braille, same as Morse. Digital displays, 5 segments instead of 7. Flag signals, mildly tricky since you only have two arms, but it can be done by mirroring the symbols so that there's 10 positions (top and bottom half), and then the bottom flag positions represent "not here". Alphabetical order is just interpreting the bits as ints. Literally any every way we have ever historically held and communicated natural language unified into a single general purpose encoding. Why settle at simplifying computer codes, when you can simplify the content at the source.
SkeuomorphicBee 3y ago
As an older programmer who deals with multiple languages, the very complexity of ASCII in a multi-language context does make unicode seems like a breath of fresh air. Seriously, multi-language in the 90s was a hell of half-baked incomplete and incompatible solutions.
cryptonector 3y ago
ASCII is a multi-byte encoding system. Yes, really, it is. The character set was designed specifically so that overstrike can be used to express a variety of characters from Latin scripts: lower-case ones with cedille or accents. E.g., to encode 'á' (á) you'd output "a<BS>'", where BS is the ASCII backspace control character. This comes from the days of typewriters, where this is how people would print accented characters and cedilles. The lower-case restriction is why Spanish makes accents optional on upper-case letters. Most of this overstrike functionality was lost long ago except for underline and bold, which remain in use in terminals (and `nroff`, and...).

franciscop 3y ago

As a beginner programmer back in the day I'd have agreed that unicode is weird and "length in characters" is the right metric for the database entry. However as a senior dev, who have read and messed around enough with binary, UTF-X, compression/GZIP, etc, I'd say that "character length" for database field size is a weird concept and that "size in bytes" would make more sense since that maps better to what you have in the HDD/SSD/network.

hans_castorp 3y ago
> I'd say that "character length" for database field size is a weird concept Until you want a constraint for such a column. From a business perspective, you would limit the column to a specific number of characters, not bytes
- bruce511 3y ago
  Let me be clear. You currently cannot constrain for number of characters in a unicode string. From a business perspective it's unhelpful to think of constraining in that way. By all means constrain on bytes, and if that's close enough for you, use that as a proxy. Or provision lots of space, and count characters in the UI.
- qalmakka 3y ago
  This is ALWAYS broken on unicode. It only makes sense in ASCII and with alphabets similar to the Latin alphabet.
  - Dylan16807 3y ago
    It's not broken if "character" means "extended grapheme cluster". Though in that case it makes it very hard to index the column.
    - qalmakka 3y ago
      Nobody cares about them, I am afraid. Most people _really_ can't accept that you can't work on Unicode like you did with ASCII, and do not want to let go of their old C habits (like iterating char by char and doing something).
- PeterisP 3y ago
  If you want a constraint for such a column, then due to nature of our complex writing, it has to be separate from the actual field type and physical storage allocation - it's the equivalent of an integer field having a constraint that it must be between 1 and 100.
- twic 3y ago
  Is there ever a genuine, non-arbitrary business requirement to limit any string to a certain number of unicode characters? And by characters here we probably mean extended grapheme clusters. If the data is going to get printed in a monospace font, on a passport or credit card or something, then i can understand a limit on the number of characters. But then it's not full unicode either - you want to constrain the length in some specific character set. Otherwise, i think limits are always arbitrary. I suspect there is a strong cultural holdover from the days of punched cards here.
HelloNurse 3y ago
Typically there should be a maximum size in bytes for performance reasons (e.g. how many rows fit in a page) and/or a minimum size in "characters" (very ill defined but usually approximated as code points, with some slack to allow a reasonable amount of combining marks in addition to the desired number of letters), with no guarantee that the two sizes are compatible. I've often found in practice short UTF-8 columns with a bytes length, that "accidentally" truncated text with ctastrophic effects after the application checked value lengths in characters.
layer8 3y ago
It makes it less intuitive for the user if you have input fields limited to n bytes instead of to n characters (complications caused by combining characters notwithstanding).
- colejohnson66 3y ago
  Twitter used(?) to have this problem: emojis, not being in the BMP, would count as two "characters" because they were encoded as UTF-16 surrogate pairs.
franciscop 3y ago
Everyone replying here seems to be quoting "business reasons" which seem very English-centric. Try to talk to a Japanese or Korean person about their "business reasons". I am suffering the opposite problem. My name has 36+ letters and I live in Japan, where the average full name has "3-4 characters" (Kanji) so they tend to be safe and allow for 10-15 characters, if I'm lucky maybe even 10 for given and 10 for family names (20 in total), where I don't have to totally murder my name and just need to amputate it.

bawolff 3y ago

> I still regularly come across folk who think unicode means 2 bytes per character. To be fair, a lot of programming interfaces with "unicode" in the name were invented when UCS-2 was the only option.

bruce511 3y ago
True. And for those of us around who "learned" unicode in the 90s, when unicode was UCS2, it can be hard to let go of those "truths that are no longer true." This complicates the learning process now.
- pastage 3y ago
  From Wikipedia. > In 1996, a surrogate character mechanism was implemented in Unicode 2.0, so that Unicode was no longer restricted to 16 bits. This increased the Unicode codespace to over a million code points, So that seems like a fair analysis. I know we still did latin-1 DBs here into 2005, but I think you had to choose charsets at that point. Everyone knew you should do Unicode but there were so many excusese not to. Unicode still is harder than ascii because of that mindset.

rcme 3y ago

None of your complaints make any sense. If you limit your usage to ASCII characters, then all of your complaints of Unicode no longer apply. E.g. character length matches your intuition of “number of characters”.

bruce511 3y ago
Firstly we live in a world where limiting yourself to ASCII characters is increasingly untenable. There are plenty of non-ascii characters in Spanish or European names. So your argument about limiting to ASCII is like saying its also not a problem if we don't use computers. Sure, you're right, but it's not terribly practical. Plus your argument uses the word Unicode (a mapping) where it only applies to utf-8, (a specific encoding ). This inaccurate use of the terms is the very issue I'm alluding to. Incidentally I'm not complaining about Unicode. Unicode is necessary and good. My comment is that (mostly older, but not exclusively) programmers find it hard to grasp the nuances of unicode and think of it as a encoding, not a mapping.
- rcme 3y ago
  > Firstly we live in a world where limiting yourself to ASCII characters is increasingly untenable. I agree. But your original comment bemoaned the difference between using ASCII and Unicode encodings. I was just pointing out that any Unicode encoding is not variable length of you limit yourself to ASCII. This applies to all Unicode encodings, not just utf-8.
  - bruce511 3y ago
    I might not have been clear in the original posting. I wasn't intending to bemoan the difference, I was bemoaning old programmers (like me) not being familiar with the most basic unicode concepts, which leads to lots of issues. Yes, if you limit your strings to ASCII then they are always 1 byte long (utf-8), or 2 bytes long (utf-16) or 4 bytes long (utf-32). So you still need to know which encoding is in play. But you can then randomly access "the 4th character". My follow up points out though that this is meaningless. Because in the real world you cannot limit your strings to just ASCII in the first place. So your point is moot from that perspective.

jesprenj 3y ago

That's why in C you have wchar_t. https://en.wikipedia.org/wiki/Wide_character https://en.wikipedia.org/wiki/Wide_character

qalmakka 3y ago
wchar_t is a massive mistake that came from an era (the '90s) where people wanted to delude themselves they could keep ASCII-like code ergonomics (iterating, random access, ...) by just changing character types. It was a stupid mistake we are still paying to this day, when OSes and libraries made in the '90s are forced to convert from the UCS2 they use internally to UTF-8 constantly (just look at Qt). The cherry on top is also that `wchar_t` is spectacularly broken in C, alongside multichar encoding and everything that's not ASCII (https://thephd.dev/cuneicode-and-the-future-of-text-in-c https://thephd.dev/cuneicode-and-the-future-of-text-in-c).
- jesprenj 3y ago
  Isn't this similar to what modern languages did? They abstracted away the underlying encoding so that the programmer deals with characters instead of bytes. Two examples are Python and Javascript strings. They kept ASCII-like code ergonomics (iterating and random access). That of course required that they separated strings and byte-like objects -- like uint8_t and wchar_t. Isn't signed char actually the culprit in modern C and therefore useless? It's main use is ASCII and that's obsolete.
  - qalmakka 3y ago
    > so that the programmer deals with characters instead of bytes This is a broken approach and a broken mindset. Modern languages like Rust (Python is not modern anymore, it is ATM 33 years old which is older than C was in 1990) reverted back to "a string is an array of uint8" because that's the only sane way to operate on them. Naive iteration and random access are broken unless they are performed on the underlying bytes, because iterating on Unicode characters is a *broken concept*. Python strings are also arguably somewhat broken, because they still allow random access into a string at "character" (not byte) indexes which causes all sorts of issues when slicing. This means that code that works perfectly with English text will malfunction when handling other languages. The hard truth is that slicing an Unicode string is a non-trivial and (somewhat) expensive operation, while Python slicing was designed in Python < 2.7 with the assumption char == byte, which now is always broken in every encoding except 8 bit, single codepage ones. For instance: >>> print(s) 'Crêpe' >>> [unicodedata.name(c) for c in s] ['LATIN CAPITAL LETTER C', 'LATIN SMALL LETTER R', 'LATIN SMALL LETTER E', 'COMBINING CIRCUMFLEX ACCENT', 'LATIN SMALL LETTER P', 'LATIN SMALL LETTER E'] >>> len(s) 6 In this case, the string "Crêpe" is in NFD form (all decomposable characters are decomposed; in particular `ê` is not U+00EA 'LATIN SMALL LETTER E WITH CIRCUMFLEX' but '\u0065\u0302', which is U+0065 'LATIN SMALL LETTER E' plus U+0302 'COMBINING CIRCUMFLEX ACCENT' ( ◌̂ ) In Rust, which is more modern, enforces UTF-8 (and not some broken version of UCS-2 or worse) slicing is ALWAYS done on bytes because it doesn't make sense to perform it on "codepoints". Asking for a slice with a byte range that falls in the middle of an UTF-8 multichar sequence will cause a panic; this also means that if you do s[2] you get the second _byte_, not "char". If you want the second character, you are forced to go through a Unicode library, as you always should. Python will instead happily comply, basically returning only the codepoints you asked, because it sees a string composed of 6 Unicode codepoints despite the fact the user only sees 5 rendered on screen: >>> s[0:3] 'Cre' >>> [unicodedata.name(c) for c in s[0:3]] ['LATIN CAPITAL LETTER C', 'LATIN SMALL LETTER R', 'LATIN SMALL LETTER E'] This makes string slicing basically useless, because even if you normalise all strings using `unicodedata.normalize('NFC', s)` before performing slicing on them, there are still several printable characters which are not represented with a single Unicode codepoint. For instance, >>> eu = '' >>> len(eu) 2 >>> x[0:1] '' because all flag emojis are represented with two Unicode codepoints, each one representing a letter. TL;DR: do not use wchar_t, UTF-16, UTF-32, ... only use UTF-8 if possible and under all circumstances treat strings a special versions as black boxes specialised for text you can only access byte per byte. If you need to do text operations, use a library like ICU or whatever your language/repository provides.

the_mitsuhiko 3y ago

Maybe but for people who worked a lot with code pages / different ISO encodings unicode brings simplicity that wasn’t there before.

nayuki 3y ago

In ASCII, the concept of a character is overloaded with many properties. Unicode breaks "character" out to separate concepts like code unit, code point, grapheme cluster, etc. Because of this, some of your statements are invalid. For example, "Along comes unicode which has variable bytes per character. (Yes, even for utf-32, which is why no-one uses utf-32).". UTF-32 has a fixed number of bytes per code point. It doesn't have a fixed number of bytes per grapheme cluster.

bruce511 3y ago
Correct. I used character where you used grapheme cluster. Utf-32 is variable length at the character (aka grapheme cluster) level. If there are any other invalid parts please elaborate - I'm always keen to learn (seriously).

qalmakka 3y ago

The gist of unicode is that code like const char *const end = str + strlen(str); for (const char *it = str; it < end; ++i) { .. } is almost always wrong unless your locale is C and you explicitly say you only work with ASCII. It's wrong with UTF-32 and char32_t too, because as the post suggests Unicode codepoints are not glyphs - you can have glyphs spanning over multiple codepoints, with multiple normalization forms. This is particularly tricky, because characters printed the same can be represented in multiple ways and it's hard to discern them, i.e. è can be a single codepoint or a combination of a '◌́ ' modifier and 'e'. In general the trivial C assumption "strings are just char arrays" most people are accustomed to is dangerous and broken in Unicode. str[x] in Unicode is wrong 99% of the time, thus UTF16 and 32 are nonsense¹ and either you treat strings as bytes you just pass around, or you need a full Unicode library such as ICU to operate on Unicode text safely. ¹: in my experience they are NEVER used mostly, UTF-16 is actually two incompatible encodings and use it as a way to "UTF-wash" the broken UCS-2 they so eagerly adopted in the '90s (including Java, Windows NT, Cocoa, Qt, Unreal, JavaScript, [partially] Python, ...) .

electroly 3y ago
UTF-16 is the internal encoding of ICU to this day. If you're using ICU, you're using UTF-16. The library treats UTF-8 as a conversion target rather than a native representation. If you ever see a new project pick UTF-16 and you don't know why, it's because of ICU; any other choice forces a round trip conversion on every ICU call. If you pick UTF-16 you can just use icu::UnicodeString as your string representation and life is easy.
johnnyanmac 3y ago
So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to iterate before I get a proper glyph instead of part of a glyph, or worse: a glyph and some extra codepoints? Is the concept so esoteric that it's best to just find a blackbox library and not worry about the finer details?
- HelloNurse 3y ago

aardvark179 3y ago

That ship sailed long before Unicode. ISO/IEC 8859-1 Included several accents which combine with a preceding character, and that was standardised in 1987. If you only deal in English then remove all accents and map everything down to 7 bit ASCII, if you don’t just deal with English then accept that and deal with it. :-) Having had to do some internationalisation of code written by Americans that couldn’t even handle 8 bit characters I am fine with spreading this bit of pain a little more evenly round the world.

dspillett 3y ago
> ISO/IEC 8859-1 … standardised in 1987 Are you sure about that designation? I don't remember 8859-1 having combining characters and other sources (https://en.wikipedia.org/wiki/ISO/IEC_8859-1 https://en.wikipedia.org/wiki/ISO/IEC_8859-1 for one) seem to agree and also list it as being officially ratified later than '87 ('98).

unconed 3y ago

UTF-8 is 30 years old, and 20 years ago it was already in common use on the web. I know because I was the doofus who pushed PHP projects to adopt it because monolinguals kept fucking up character encodings. Being an older programmer is a very poor excuse at this point to not understand Unicode.

bruce511 3y ago
>> Being an older programmer is a very poor excuse at this point to not understand Unicode. I completely agree. Which does not make the ignorance I experience around Unicode from both older, and younger programmers, easier to forgive.

euroderf 3y ago

> the very simplicity of ASCII can make unicode difficult to grasp. Somewhat related: I hate "smart" "quotes". I find myself editing them into plain old << " >> and << ' >>. If you want to use "smart" quotes to add semantic meaning to text - as in "this is a quote" - then you should be using markup to do the job. Stop abusing non-ASCII typography so you can pretend to write "semantic" material that you think will impress all the cool kids that know structured documentation. Who else remembers that great Perl script, "The Demoronizer" ?

chrismorgan 3y ago
I too hate “smart quotes”, meaning anything that automatically converts from ASCII, but only for myself. That’s because I’m a weirdo that types exactly what I mean, things like em dashes and en dashes and curly quotes and such, using my Compose key. Opening and closing quotation marks are different from one another (simple fact, true from the very early days of quotation marks), so they should be represented differently in Unicode (simple fact). Anything along the lines of “smart quotes” is just a pragmatic approach to let normal people get the correct quotation marks, and I’m mostly glad they can have it (I just wish it’d handle things like “’cos” and “’90s” properly, which use an apostrophe which is equivalent to a right single quotation mark; but other than that, I’m glad when people use curly quotes rather than straight quotes), so long as I don’t have to have it (and I’m glad to say I don’t).

JdeBP 3y ago

Older programmers like you are not old enough. (-: ASCII had variable numbers of bytes per glyph back in the 1960s, and the Teletype Model 37 semantics of combining-backspace and combining-underscore still exist in modern operating systems, like the various BSDs and Linux-based ones, to this day. It's a combining-characters multibyte encoding understood by programs from less(1) through ul(1) to nroff(1). ASCII had multiple transfer encodings, from 8N1 to 7E2, many of which were neither 7-bit nor 8-bit; and 1 byte of RAM was not 1 ASCII character. 1 byte of RAM not being 1 ASCII character is, after all, what brought about "extended ASCII". Along with that came ECMA-35, with its GR and GL, and code pages. All of those broke theretofore held truths such as that one could just strip the parity bit, actually introduced the false idea (that many people had to keep correcting) that ASCII was 8-bit, and required a full-blown state machine for parsing ECMA-35 escape sequences. Then there are the tales that old programmers can tell about how they couldn't actually deal in English because ASCII didn't have a pound symbol, and what was "English" was actually American (which was, after all, the "A" in "ASCII"). There is a whole generation of older programmers who laugh hollowly at the idea of "the simplicity of ASCII"; when the amount of time that they spent fiddling with serial protocol settings, transcoding between code pages, handling language variant characters in Teletext, wondering why an 8-bit-clean transport protocol such as TCP didn't mean that SMTP could be 8-bit clean, and doing other such stuff, probably came to years in total spent on all of this "simplicity".

msla 3y ago
> Then there are the tales that old programmers can tell about how they couldn't actually deal in English because ASCII didn't have a pound symbol What's #, then? Chopped liver? /s
- pwdisswordfishc 3y ago
  The number sign, of course.

IIAOPSW 3y ago

You don't go far enough. Lets do some language reform. The entire alphabet is always just 5 bits expressed in different mediums. Lets make the written characters are all based on 5 sticks coming out of a central point literally representing your hand. Sign language, pretty obvious. Morse code, the up / down fingers directly translate to holding the key down or not. Braille, same as Morse. Digital displays, 5 segments instead of 7. Flag signals, mildly tricky since you only have two arms, but it can be done by mirroring the symbols so that there's 10 positions (top and bottom half), and then the bottom flag positions represent "not here". Alphabetical order is just interpreting the bits as ints. Literally any every way we have ever historically held and communicated natural language unified into a single general purpose encoding. Why settle at simplifying computer codes, when you can simplify the content at the source.

SkeuomorphicBee 3y ago

As an older programmer who deals with multiple languages, the very complexity of ASCII in a multi-language context does make unicode seems like a breath of fresh air. Seriously, multi-language in the 90s was a hell of half-baked incomplete and incompatible solutions.

cryptonector 3y ago

ASCII is a multi-byte encoding system. Yes, really, it is. The character set was designed specifically so that overstrike can be used to express a variety of characters from Latin scripts: lower-case ones with cedille or accents. E.g., to encode 'á' (á) you'd output "a<BS>'", where BS is the ASCII backspace control character. This comes from the days of typewriters, where this is how people would print accented characters and cedilles. The lower-case restriction is why Spanish makes accents optional on upper-case letters. Most of this overstrike functionality was lost long ago except for underline and bold, which remain in use in terminals (and `nroff`, and...).