11 ms·

UTF-21, a toy character encoding

93 points by samwillis 3y ago

recursive 3y ago
If you exclude surrogates, I think you can fit the rest of the codepoints in 20 bits. There would be no need for surrogates all Unicode scalars are representable elseways. Looks like an easy size win.
- Sniffnoy 3y ago
  This is not true; the number of Unicode scalar values excluding surrogates is 17*2^16 - 2^11 = 1,112,064, which is larger than 2^20 = 1,048,576.
  - nayuki 3y ago
    In particular, UTF-16 surrogates use 2048 code points in the basic multilingual plane (BMP), but surrogates allow another 2^20 code points to be encodable in addition to the non-surrogate 65536−2048 code points of the BMP.
CGamesPlay 3y ago
I'd appreciate an analysis of how it compresses! The encoding looks highly compressible, so I'd expect it to be competitive with UTF-8 for English text, and seems like it would beat it for East Asian languages.
- bruce511 3y ago
  At first glance I'd suggest it doesn't compress at all. Especially if the compression uses bytes. Bit-streams of characters of non-bytes length become randomish when viewed as bytes, and random bytes contain no redundancy and can't be compressed.
  - Joker_vD 3y ago
    > Especially if the compression uses bytes Arithmetic compression can use whatever, even fractional bits, and it's been around since the seventies.
    - Timwi 3y ago
      Yes, but if you run a normal compression algorithm like gz, rar or 7z on it, it's still going to use bytes.
    - temac 3y ago
      Arithmetic compression uses whatever on the output. Of course you can retokenize weird input but you can usually do so for any algo if you can modify it. But UTF21 can not have a substantial advantage if you compress. It will usually be worse.
    - Dylan16807 3y ago
      Arithmetic coding goes one token/symbol at a time, just like most kinds of compression. The fractional bits come after token selection, and aren't really relevant here. You can split the input into tokens that aren't a multiple of 8 bits, sure. But that's its own decision. 7 or 21 or whatever bit tokens could be fed into a huffman tree just as easily.
bruce511 3y ago
For older programmers like me, who pretty much only deal in English, the very simplicity of ASCII can make unicode difficult to grasp. In ASCII 1 byte (always 1 byte) of ram = 1 character. And the encoding called ASCII matches the mapping called ASCII. Along comes Unicode (a mapping), which has multiple different encodings. A difficult distinction leading to statements like "that's a unicode string/file/field". Along comes unicode which has variable bytes per character. (Yes, even for utf-32, which is why no-one uses utf-32). I still regularly come across folk who think unicode means 2 bytes per character. Along comes unicode which asks you to consider if the functions LEN, SUB, SLICE etc are counting Bytes or Characters. Along comes unicode that breaks the idea of setting field length in databases as "number of characters", made worse by a cohort brought up on limited storage, who want to be "effecient" in their declarations. Along comes unicode with code units, code points, characters -all of which come into play, all with different, or variable, lengths. So while utf-21 might be a "toy", even making something like that yourself is a learning exercise well worth the endeavour. It should be a mandatory teaching exercise. Two thumbs up.
- franciscop 3y ago
  As a beginner programmer back in the day I'd have agreed that unicode is weird and "length in characters" is the right metric for the database entry. However as a senior dev, who have read and messed around enough with binary, UTF-X, compression/GZIP, etc, I'd say that "character length" for database field size is a weird concept and that "size in bytes" would make more sense since that maps better to what you have in the HDD/SSD/network.
  - hans_castorp 3y ago
    > I'd say that "character length" for database field size is a weird concept Until you want a constraint for such a column. From a business perspective, you would limit the column to a specific number of characters, not bytes
    - bruce511 3y ago
      Let me be clear. You currently cannot constrain for number of characters in a unicode string. From a business perspective it's unhelpful to think of constraining in that way. By all means constrain on bytes, and if that's close enough for you, use that as a proxy. Or provision lots of space, and count characters in the UI.
skissane 3y ago
> Unicode has three official ways of storing these numbers: UTF-8, UTF-16, and UTF-32 They are less official, but Unicode defines several other encodings in their technical reports and notes: SCSU: https://www.unicode.org/reports/tr6/tr6-4.html https://www.unicode.org/reports/tr6/tr6-4.html UTF-EBCDIC: http://www.unicode.org/reports/tr16/tr16-8.html http://www.unicode.org/reports/tr16/tr16-8.html CESU-8: https://www.unicode.org/reports/tr26/tr26-4.html https://www.unicode.org/reports/tr26/tr26-4.html BOCU-1: https://www.unicode.org/notes/tn6/ https://www.unicode.org/notes/tn6/ These are all somewhat legacy, having never seen very wide adoption: Compression schemes such as SCSU and BOCU-1 turned out in practice to add a lot of complexity for minimal real-world benefit. If you really need to save storage, then a standard compression algorithm (e.g. deflate or zstd) is usually a better option. CESU-8 is a backward compatibility hack for old UTF-8 implementations which used encoded UTF-16 surrogates for codepoints outside the BMP. UTF-EBCDIC is basically UTF-8 but for EBCDIC. It was invented by someone at IBM, but IBM themselves ended up standardising on UTF-16 instead. And if IBM ended up not using UTF-EBCDIC, who else was going to? Well, there's a few surviving non-IBM mainframe vendors who also use EBCDIC (such as Unisys and Fujitsu BS2000), but I'm not aware any of them ever expressed any interest in it either.
- wpietri 3y ago
  > UTF-EBCDIC Just reading that made my eyelid twitch.
  - ElectricalUnion 3y ago
    Almost nothing uses UTF-EBCDIC by default, db2 (and the underlying VSAM persistent storage) mostly ends up writing UTF-16 (but you can set it up to write UTF-8); except for internal catalog data and network traffic, that are usually UTF-8 no matter what.
    - skissane 3y ago
      Do any IBM products have UTF-EBCDIC as a supported configuration option? Does anyone use it? Oracle RDBMS supports "UTFE" on EBCDIC platforms, which is to UTF-EBCDIC what CESU-8 is to UTF-8 – i.e. UTF-EBCDIC but with codepoints outside the BMP encoded using surrogates. Over the last couple of decades, the only EBCDIC platforms Oracle RDBMS has been supported on have been z/OS and BS2000/OSD – once upon a time it was also available on VM/CMS (not sure when that was discontinued), and possibly more besides (in the 80s thru early 90s, Oracle ported their DB to just about everything under the sun, they later became much more selective in what they'd support). The z/OS port was discontinued with version 10gR2 (initial release 2005, final patchset 2010) – although, while Oracle normally puts a time limit on their pay-extra support for patches for old versions ("extended support"), exceptionally they've said for the z/OS port they'll continue offering that as long as customers want it. So the only remaining EBCDIC port of Oracle RDBMS is Fujitsu BS2000/OSD – I think the only reason that port survives is because Fujitsu pays Oracle to keep on producing it. mksh (MirBSD's ksh) supports some custom EBCDIC encoding of UTF-8 (so-called "nega-UTF-8"), instead of proper UTF-EBCDIC. One difference is that UTF-EBCDIC avoids using the C1 controls (bytes 0x80 thru 0x9F), so EBCDIC control characters can be left intact, whereas "nega-UTF-8" just converts UTF-8 to EBCDIC using an arbitrary EBCDIC code page, hence failing to fully preserve the full complement of EBCDIC control characters. Also, while UTF-EBCDIC is based on a fixed EBCDIC code page (1047), "nega-UTF-8" isn't, so it really isn't a single encoding, rather a family of them. Has anyone else ever implemented UTF-EBCDIC, or something similar (given neither Oracle UTFE nor mksh's nega-UTF-8 are actually UTF-EBCDIC)? Years ago, I wrote an implementation of it myself (in Java), which I've never released – not for any work-related purpose, just as a private exercise in recreational programming. Actually, I have an FTP server I wrote which serves up UTF-EBCDIC as "TYPE E", but I could never test it properly, because (at the time) I couldn't find an FTP client which actually implemented TYPE E. I think I later found one, but by then I'd lost interest in pursuing it any further.
userbinator 3y ago
I really don't see any advantage to UTF-32 when not-officially-standardised UTF-24 has the same constant 3-byte-sized codepoints (and multiplying by 3 is not hard - it's n + 2n); in UTF-32, the highest byte will never be anything other than 0, so it's essentially permanent waste. Also, according to https://en.wikipedia.org/wiki/List_of_Unicode_characters https://en.wikipedia.org/wiki/List_of_Unicode_characters there's currently less than 150k codepoints defined, so even 21 bits is several times larger than necessary --- 18 bits will contain all the currently assigned codepoints, and be sufficient until 256k codepoints is reached. Incidentally, 18-bit architectures were once common: https://en.wikipedia.org/wiki/18-bit_computing https://en.wikipedia.org/wiki/18-bit_computing
- Findecanor 3y ago
  To fit all currently assigned code points within 18 bits is easy: you would only have to move one range. Above 32FFF, all assigned code points are within E0000 to E01EF, which fits in-between 32FFF to 3FFFF with room to spare. Those code points are used for flag emojis and for selecting uncommon CJK variants. If you don't support those, you could just strip out anything that doesn't fit in 18 bits to begin with.
- skissane 3y ago
  > I really don't see any advantage to UTF-32 when not-officially-standardised UTF-24 has the same constant 3-byte-sized codepoints (and multiplying by 3 is not hard - it's n + 2n); in UTF-32, the highest byte will never be anything other than 0, so it's essentially permanent waste. Wouldn’t the difference in alignment (4 byte versus 3 bytes) make UTF-32 faster than UTF-24 in certain cases, on certain CPU architectures? So the always zero byte would be wasting space to gain greater performance.
- electroly 3y ago
  UTF-32 is easy to understand for educational purposes but it's probably a mistake to use it as a real string representation, and almost nobody does. Code units and code points are the same thing in UTF-32 but they're different in UTF-16 and UTF-8; you can teach someone UTF-32 before they understand the distinction. Obviously, UTF-24 isn't used because it isn't a standard encoding, and if you really wanted to save memory, you'd use UTF-8 instead which is even more compact yet. As for UTF-16, today the only reason people choose UTF-16 for new projects is because it's the native internal encoding of the ICU library. If you're not using ICU, it's pretty hard to defend anything but UTF-8.
Timwi 3y ago
Shameless plug: I came up with UTF-21 for my esoteric programming language, Funciton (https://esolangs.org/wiki/Funciton https://esolangs.org/wiki/Funciton), back in 2011.
- LegionMammal978 3y ago
  It's funny: for all I knew when reading the post, UTF-21 was something I'd never seen before. But when I saw your comment, I remembered that I must have been quite aware of it back in 2015, when I golfed a Funciton Hello World program [0]. Seven years is a long time for such a silly thing to circle around again. [0] https://codegolf.stackexchange.com/a/55777 https://codegolf.stackexchange.com/a/55777
eviks 3y ago
Out of curiosity, is there a more compact alternative to Unicode that excludes all precomposed chars like ñ, but also for all the Asian languages, so you'd basically have some set of "strokes" that can compose anything else? And if you need to add semantic meaning to identically looking results, that would be another non-visual composable indicator
- Dylan16807 3y ago
  What's your definition of "compact" here? And there's a lot of ways to put together a character, so I worry you'd end up with something closer to a vector format than a normal character encoding.
  - eviks 3y ago
    > What's your definition of "compact" here? Number of elements in an encoding 5: ˜ n o ñ õ 3: ˜ n o > so I worry you'd end up with something closer to a vector format than a normal character encoding. then stop 1 step before that :)
    - lifthrasiir 3y ago
      Much easier said than done. Try to actually do that. A good start would be IDS.TXT I've linked above; this gives about 500 approximately atomic characters to start with [1]. Now extend this to the entirety of Ideographic Variation Database [2], which tries to solve the most cited problems with Han unification. And then add some kind of semantic annotation as you've suggested (I have no idea, maybe you have a better idea). [1] 123 components that are already encoded, 121 components ({01}, ...) that are partial characters, ~123 "unpresentable" components (？), and ~115 "minor" variations (〾) that may require additional components or two. [2] https://unicode.org/ivd/ https://unicode.org/ivd/
      - eviks 3y ago
        > Much easier said than done. Try to actually do that Ha-ha, thanks for the offer, but I'll pass - all that complexity is precisely why I'm not even going to try, it requires a lot of dedicated team effort put into it, a simple "try" is doomed to fail.
Dylan16807 3y ago
I feel like the obvious 21 bit encoding is to pack 3 codepoints into 8 bytes. But uh I guess this works. If you want to do something more in depth and make real decisions about encoding, a good space to explore is 1-3 byte variable width encodings. You have lots of different tradeoffs to consider and you can make something surprisingly good.
ttflee 3y ago
Now for an encoding in real life: https://en.wikipedia.org/wiki/UTF-7 https://en.wikipedia.org/wiki/UTF-7
- jeroenhd 3y ago
  With UTF-7 being a requirement for certain (outdated) email protocols and programs intentionally sabotaging their UTF-7 code paths (most importantly .NET 5), UTF-7 will remain relevant for anyone dealing with email software for a while. UTF-7 is just one of the many ways in which email can screw you over. SMTP can't reliably handle UTF-8 so your options quickly devolve into base64 encoding messages or falling back to UTF-7 encoding. Base64 is a lot more wasteful than the alternative, as cursed as UTF-7 may be. For full interoperability, some mail servers even require you to be able to deal with UTF-7 wrapped inside another transcoding format!
hlandau 3y ago
The Unicode space used to be 31-bit. UTF-8 defined 5 and 6-byte encoding sequences for this reason. When UCS-2 had to be extended to create UTF-16, the solution chosen effectively limited things to a 21-bit space. Unicode was then redefined to be 21-bit, and the 5 and 6-byte encodings of UTF-8 were removed from the standard, though old systems might still accept them. It is worth noting that if we ever run out of space in 21-bit Unicode, one could conceivably re-extend it back to the original 31 bits by throwing UTF-16 under the bus. The only reason Unicode is limited to 21 bits is because of UTF-16.
- teaearlgraycold 3y ago
  But JS is UTF-16 native. Would it even be feasible to migrate?
  - lifthrasiir 3y ago
    "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. JS implementations already do not use UTF-16 as a sole native representation anyway, so the migration should be relatively easy.
    - chrismorgan 3y ago
      > JS implementations already do not use UTF-16 as a sole native representation anyway But they still expose UTF-16 code unit semantics: U+10000–U+10FFFF come through as two UTF-16 code units, a surrogate pair. > "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. True enough. You could redefine it to have UTF-16 code unit semantics until U+10FFFF, then code point semantics beyond, maybe call this “UTF-16+ code unit semantics”. It’d definitely break a few things here and there, but then, anything would break a few things here and there, regardless of encoding type, since the range U+0000–U+10FFFF is hard-coded in so may places. Well, anything except doing surrogates again. And I’m pretty sure that’s what would be done.
      - 3y ago
layer8 3y ago
One aspect the article is omitting is that UTF-16 and UTF-32 aren’t strictly speaking encodings, but UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE are. Furthermore, with a 21-bit packed encoding like UTF-21, not only byte-level endianness is relevant, but inner-byte bit-level endianness becomes relevant as well. While the encoding examples given imply a particular endianness, this is something that would have to be specified more explicitly.
- chrismorgan 3y ago
  Endianness is such a pain. I wish everything had always stayed big-endian, the generally superior option (though I will acknowledge it’s not quite uniformly superior). Implement any bit-level algorithms and endianness tends to rear its ugly head. I implemented the Speck cipher last year, and it basically mixed little- and big-endianness (byte/bit expression), but the paper never mentioned this at all, so that trial and error on the sample vectors, or emailing the authors for clarification, was the only way you’d find that out! The article gives examples that assume big-endianness, but in practice, UTF-16 is almost always UTF-16LE (no idea about UTF-32 since it’s not used so much), so the bytes within each code unit are, for practical purposes, back to front.
WorldMaker 3y ago
From the article: > I think UTF-32 is the simplest. Each number is put into a 32-bit integer, or 4 bytes. This is called a “fixed-width” encoding. This is wrong. The 4-byte fixed encoding is UCS-4, and while UCS-4 currently encodes all of Unicode, there's no guarantee it will "forever". Just as people who assumed UCS-2 was "all" of Unicode have since been proven (very) wrong. UTF-32 is a variable width encoding even if right now there's no reason to use surrogates out to the next plane (particularly because the next plane is entirely inaccessible to UTF-16 and only UTF-16, but UTF-16 is still one of the world's most common encodings).
- e4m2 3y ago
  UTF-32 is a fixed-width encoding. To quote directly from the Unicode standard, chapter 3.9, definition 90: > UTF-32 encoding form: The Unicode encoding form that assigns each Unicode scalar value to a single unsigned 32-bit code unit with the same numeric value as the Unicode scalar value. UCS-4 is defined in ISO 10646. Before the 2011 revision of the standard UCS-4 had a 31-bit codespace, since then the definitions of the ISO and Unicode standards have converged and UCS-4 and UTF-32 are now synonymous. > while UCS-4 currently encodes all of Unicode, there's no guarantee it will "forever". There is. Unicode has a codespace of U+0000..U+10FFFF. There are 2^16 + 2^20 − 2^11 assignable codepoints, in Unicode 15 a total of 149,186 (13.4%) have been assigned. Even if the Unicode codespace were to ever be extended again (it won't), the only encoding that would become incompatible is UTF-16. In fact, both UTF-8 and UTF-32 are trivially extensible and used to be wider encodings, but were restricted to 0x10FFFF arbitrarily to match UTF-16 limitations. EDIT: My wording is very ambiguous. There are actually 286,719 (25.8%) assigned codepoints in Unicode 15: https://www.unicode.org/versions/stats/charcountv15_0.html https://www.unicode.org/versions/stats/charcountv15_0.html. Still, the point stands.
  - chrismorgan 3y ago
    > Even if the Unicode codespace were to ever be extended again (it won't), the only encoding that would become incompatible is UTF-16. In fact, both UTF-8 and UTF-32 are trivially extensible and used to be wider encodings, but were restricted to 0x10FFFF arbitrarily to match UTF-16 limitations. Mind you, it wouldn’t be this easy, because things should perform Unicode validation, and many do, so every piece of software would need to be updated for the new, enlarged version of Unicode, UTF-8 and UTF-32, and old software that validated would baulk or convert anything from the new ranges into REPLACEMENT CHARACTER. But yeah, UTF-16 would be toast.
OscarCunningham 3y ago
Would Huffman coding be called UTF-1?
erichocean 3y ago
The Factor language implemented Unicode support this way.

93 points by samwillis 3y ago

If you exclude surrogates, I think you can fit the rest of the codepoints in 20 bits. There would be no need for surrogates all Unicode scalars are representable elseways. Looks like an easy size win.

Sniffnoy 3y ago
This is not true; the number of Unicode scalar values excluding surrogates is 17*2^16 - 2^11 = 1,112,064, which is larger than 2^20 = 1,048,576.
- nayuki 3y ago
  In particular, UTF-16 surrogates use 2048 code points in the basic multilingual plane (BMP), but surrogates allow another 2^20 code points to be encodable in addition to the non-surrogate 65536−2048 code points of the BMP.

CGamesPlay 3y ago

I'd appreciate an analysis of how it compresses! The encoding looks highly compressible, so I'd expect it to be competitive with UTF-8 for English text, and seems like it would beat it for East Asian languages.

bruce511 3y ago
At first glance I'd suggest it doesn't compress at all. Especially if the compression uses bytes. Bit-streams of characters of non-bytes length become randomish when viewed as bytes, and random bytes contain no redundancy and can't be compressed.
- Joker_vD 3y ago
  > Especially if the compression uses bytes Arithmetic compression can use whatever, even fractional bits, and it's been around since the seventies.
  - Timwi 3y ago
    Yes, but if you run a normal compression algorithm like gz, rar or 7z on it, it's still going to use bytes.
  - temac 3y ago
    Arithmetic compression uses whatever on the output. Of course you can retokenize weird input but you can usually do so for any algo if you can modify it. But UTF21 can not have a substantial advantage if you compress. It will usually be worse.
  - Dylan16807 3y ago
    Arithmetic coding goes one token/symbol at a time, just like most kinds of compression. The fractional bits come after token selection, and aren't really relevant here. You can split the input into tokens that aren't a multiple of 8 bits, sure. But that's its own decision. 7 or 21 or whatever bit tokens could be fed into a huffman tree just as easily.

bruce511 3y ago

For older programmers like me, who pretty much only deal in English, the very simplicity of ASCII can make unicode difficult to grasp. In ASCII 1 byte (always 1 byte) of ram = 1 character. And the encoding called ASCII matches the mapping called ASCII. Along comes Unicode (a mapping), which has multiple different encodings. A difficult distinction leading to statements like "that's a unicode string/file/field". Along comes unicode which has variable bytes per character. (Yes, even for utf-32, which is why no-one uses utf-32). I still regularly come across folk who think unicode means 2 bytes per character. Along comes unicode which asks you to consider if the functions LEN, SUB, SLICE etc are counting Bytes or Characters. Along comes unicode that breaks the idea of setting field length in databases as "number of characters", made worse by a cohort brought up on limited storage, who want to be "effecient" in their declarations. Along comes unicode with code units, code points, characters -all of which come into play, all with different, or variable, lengths. So while utf-21 might be a "toy", even making something like that yourself is a learning exercise well worth the endeavour. It should be a mandatory teaching exercise. Two thumbs up.

franciscop 3y ago
As a beginner programmer back in the day I'd have agreed that unicode is weird and "length in characters" is the right metric for the database entry. However as a senior dev, who have read and messed around enough with binary, UTF-X, compression/GZIP, etc, I'd say that "character length" for database field size is a weird concept and that "size in bytes" would make more sense since that maps better to what you have in the HDD/SSD/network.
- hans_castorp 3y ago
  > I'd say that "character length" for database field size is a weird concept Until you want a constraint for such a column. From a business perspective, you would limit the column to a specific number of characters, not bytes
  - bruce511 3y ago
    Let me be clear. You currently cannot constrain for number of characters in a unicode string. From a business perspective it's unhelpful to think of constraining in that way. By all means constrain on bytes, and if that's close enough for you, use that as a proxy. Or provision lots of space, and count characters in the UI.

skissane 3y ago

> Unicode has three official ways of storing these numbers: UTF-8, UTF-16, and UTF-32 They are less official, but Unicode defines several other encodings in their technical reports and notes: SCSU: https://www.unicode.org/reports/tr6/tr6-4.html https://www.unicode.org/reports/tr6/tr6-4.html UTF-EBCDIC: http://www.unicode.org/reports/tr16/tr16-8.html http://www.unicode.org/reports/tr16/tr16-8.html CESU-8: https://www.unicode.org/reports/tr26/tr26-4.html https://www.unicode.org/reports/tr26/tr26-4.html BOCU-1: https://www.unicode.org/notes/tn6/ https://www.unicode.org/notes/tn6/ These are all somewhat legacy, having never seen very wide adoption: Compression schemes such as SCSU and BOCU-1 turned out in practice to add a lot of complexity for minimal real-world benefit. If you really need to save storage, then a standard compression algorithm (e.g. deflate or zstd) is usually a better option. CESU-8 is a backward compatibility hack for old UTF-8 implementations which used encoded UTF-16 surrogates for codepoints outside the BMP. UTF-EBCDIC is basically UTF-8 but for EBCDIC. It was invented by someone at IBM, but IBM themselves ended up standardising on UTF-16 instead. And if IBM ended up not using UTF-EBCDIC, who else was going to? Well, there's a few surviving non-IBM mainframe vendors who also use EBCDIC (such as Unisys and Fujitsu BS2000), but I'm not aware any of them ever expressed any interest in it either.

wpietri 3y ago
> UTF-EBCDIC Just reading that made my eyelid twitch.
- ElectricalUnion 3y ago
  Almost nothing uses UTF-EBCDIC by default, db2 (and the underlying VSAM persistent storage) mostly ends up writing UTF-16 (but you can set it up to write UTF-8); except for internal catalog data and network traffic, that are usually UTF-8 no matter what.
  - skissane 3y ago
    Do any IBM products have UTF-EBCDIC as a supported configuration option? Does anyone use it? Oracle RDBMS supports "UTFE" on EBCDIC platforms, which is to UTF-EBCDIC what CESU-8 is to UTF-8 – i.e. UTF-EBCDIC but with codepoints outside the BMP encoded using surrogates. Over the last couple of decades, the only EBCDIC platforms Oracle RDBMS has been supported on have been z/OS and BS2000/OSD – once upon a time it was also available on VM/CMS (not sure when that was discontinued), and possibly more besides (in the 80s thru early 90s, Oracle ported their DB to just about everything under the sun, they later became much more selective in what they'd support). The z/OS port was discontinued with version 10gR2 (initial release 2005, final patchset 2010) – although, while Oracle normally puts a time limit on their pay-extra support for patches for old versions ("extended support"), exceptionally they've said for the z/OS port they'll continue offering that as long as customers want it. So the only remaining EBCDIC port of Oracle RDBMS is Fujitsu BS2000/OSD – I think the only reason that port survives is because Fujitsu pays Oracle to keep on producing it. mksh (MirBSD's ksh) supports some custom EBCDIC encoding of UTF-8 (so-called "nega-UTF-8"), instead of proper UTF-EBCDIC. One difference is that UTF-EBCDIC avoids using the C1 controls (bytes 0x80 thru 0x9F), so EBCDIC control characters can be left intact, whereas "nega-UTF-8" just converts UTF-8 to EBCDIC using an arbitrary EBCDIC code page, hence failing to fully preserve the full complement of EBCDIC control characters. Also, while UTF-EBCDIC is based on a fixed EBCDIC code page (1047), "nega-UTF-8" isn't, so it really isn't a single encoding, rather a family of them. Has anyone else ever implemented UTF-EBCDIC, or something similar (given neither Oracle UTFE nor mksh's nega-UTF-8 are actually UTF-EBCDIC)? Years ago, I wrote an implementation of it myself (in Java), which I've never released – not for any work-related purpose, just as a private exercise in recreational programming. Actually, I have an FTP server I wrote which serves up UTF-EBCDIC as "TYPE E", but I could never test it properly, because (at the time) I couldn't find an FTP client which actually implemented TYPE E. I think I later found one, but by then I'd lost interest in pursuing it any further.

userbinator 3y ago

I really don't see any advantage to UTF-32 when not-officially-standardised UTF-24 has the same constant 3-byte-sized codepoints (and multiplying by 3 is not hard - it's n + 2n); in UTF-32, the highest byte will never be anything other than 0, so it's essentially permanent waste. Also, according to https://en.wikipedia.org/wiki/List_of_Unicode_characters https://en.wikipedia.org/wiki/List_of_Unicode_characters there's currently less than 150k codepoints defined, so even 21 bits is several times larger than necessary --- 18 bits will contain all the currently assigned codepoints, and be sufficient until 256k codepoints is reached. Incidentally, 18-bit architectures were once common: https://en.wikipedia.org/wiki/18-bit_computing https://en.wikipedia.org/wiki/18-bit_computing

Findecanor 3y ago
To fit all currently assigned code points within 18 bits is easy: you would only have to move one range. Above 32FFF, all assigned code points are within E0000 to E01EF, which fits in-between 32FFF to 3FFFF with room to spare. Those code points are used for flag emojis and for selecting uncommon CJK variants. If you don't support those, you could just strip out anything that doesn't fit in 18 bits to begin with.
skissane 3y ago
> I really don't see any advantage to UTF-32 when not-officially-standardised UTF-24 has the same constant 3-byte-sized codepoints (and multiplying by 3 is not hard - it's n + 2n); in UTF-32, the highest byte will never be anything other than 0, so it's essentially permanent waste. Wouldn’t the difference in alignment (4 byte versus 3 bytes) make UTF-32 faster than UTF-24 in certain cases, on certain CPU architectures? So the always zero byte would be wasting space to gain greater performance.
electroly 3y ago
UTF-32 is easy to understand for educational purposes but it's probably a mistake to use it as a real string representation, and almost nobody does. Code units and code points are the same thing in UTF-32 but they're different in UTF-16 and UTF-8; you can teach someone UTF-32 before they understand the distinction. Obviously, UTF-24 isn't used because it isn't a standard encoding, and if you really wanted to save memory, you'd use UTF-8 instead which is even more compact yet. As for UTF-16, today the only reason people choose UTF-16 for new projects is because it's the native internal encoding of the ICU library. If you're not using ICU, it's pretty hard to defend anything but UTF-8.

Timwi 3y ago

Shameless plug: I came up with UTF-21 for my esoteric programming language, Funciton (https://esolangs.org/wiki/Funciton https://esolangs.org/wiki/Funciton), back in 2011.

LegionMammal978 3y ago
It's funny: for all I knew when reading the post, UTF-21 was something I'd never seen before. But when I saw your comment, I remembered that I must have been quite aware of it back in 2015, when I golfed a Funciton Hello World program [0]. Seven years is a long time for such a silly thing to circle around again. [0] https://codegolf.stackexchange.com/a/55777 https://codegolf.stackexchange.com/a/55777

eviks 3y ago

Out of curiosity, is there a more compact alternative to Unicode that excludes all precomposed chars like ñ, but also for all the Asian languages, so you'd basically have some set of "strokes" that can compose anything else? And if you need to add semantic meaning to identically looking results, that would be another non-visual composable indicator

Dylan16807 3y ago
What's your definition of "compact" here? And there's a lot of ways to put together a character, so I worry you'd end up with something closer to a vector format than a normal character encoding.
- eviks 3y ago
  > What's your definition of "compact" here? Number of elements in an encoding 5: ˜ n o ñ õ 3: ˜ n o > so I worry you'd end up with something closer to a vector format than a normal character encoding. then stop 1 step before that :)
  - lifthrasiir 3y ago
    Much easier said than done. Try to actually do that. A good start would be IDS.TXT I've linked above; this gives about 500 approximately atomic characters to start with [1]. Now extend this to the entirety of Ideographic Variation Database [2], which tries to solve the most cited problems with Han unification. And then add some kind of semantic annotation as you've suggested (I have no idea, maybe you have a better idea). [1] 123 components that are already encoded, 121 components ({01}, ...) that are partial characters, ~123 "unpresentable" components (？), and ~115 "minor" variations (〾) that may require additional components or two. [2] https://unicode.org/ivd/ https://unicode.org/ivd/
    - eviks 3y ago
      > Much easier said than done. Try to actually do that Ha-ha, thanks for the offer, but I'll pass - all that complexity is precisely why I'm not even going to try, it requires a lot of dedicated team effort put into it, a simple "try" is doomed to fail.

Dylan16807 3y ago

I feel like the obvious 21 bit encoding is to pack 3 codepoints into 8 bytes. But uh I guess this works. If you want to do something more in depth and make real decisions about encoding, a good space to explore is 1-3 byte variable width encodings. You have lots of different tradeoffs to consider and you can make something surprisingly good.

ttflee 3y ago

Now for an encoding in real life: https://en.wikipedia.org/wiki/UTF-7 https://en.wikipedia.org/wiki/UTF-7

jeroenhd 3y ago
With UTF-7 being a requirement for certain (outdated) email protocols and programs intentionally sabotaging their UTF-7 code paths (most importantly .NET 5), UTF-7 will remain relevant for anyone dealing with email software for a while. UTF-7 is just one of the many ways in which email can screw you over. SMTP can't reliably handle UTF-8 so your options quickly devolve into base64 encoding messages or falling back to UTF-7 encoding. Base64 is a lot more wasteful than the alternative, as cursed as UTF-7 may be. For full interoperability, some mail servers even require you to be able to deal with UTF-7 wrapped inside another transcoding format!

hlandau 3y ago

The Unicode space used to be 31-bit. UTF-8 defined 5 and 6-byte encoding sequences for this reason. When UCS-2 had to be extended to create UTF-16, the solution chosen effectively limited things to a 21-bit space. Unicode was then redefined to be 21-bit, and the 5 and 6-byte encodings of UTF-8 were removed from the standard, though old systems might still accept them. It is worth noting that if we ever run out of space in 21-bit Unicode, one could conceivably re-extend it back to the original 31 bits by throwing UTF-16 under the bus. The only reason Unicode is limited to 21 bits is because of UTF-16.

teaearlgraycold 3y ago
But JS is UTF-16 native. Would it even be feasible to migrate?
- lifthrasiir 3y ago
  "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. JS implementations already do not use UTF-16 as a sole native representation anyway, so the migration should be relatively easy.
  - chrismorgan 3y ago
    > JS implementations already do not use UTF-16 as a sole native representation anyway But they still expose UTF-16 code unit semantics: U+10000–U+10FFFF come through as two UTF-16 code units, a surrogate pair. > "UTF-16 native" doesn't mean your "UTF-16 unit" (i.e. what would `charCodeAt` return) should not exceed 16 bits. True enough. You could redefine it to have UTF-16 code unit semantics until U+10FFFF, then code point semantics beyond, maybe call this “UTF-16+ code unit semantics”. It’d definitely break a few things here and there, but then, anything would break a few things here and there, regardless of encoding type, since the range U+0000–U+10FFFF is hard-coded in so may places. Well, anything except doing surrogates again. And I’m pretty sure that’s what would be done.
    - 3y ago

layer8 3y ago

One aspect the article is omitting is that UTF-16 and UTF-32 aren’t strictly speaking encodings, but UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE are. Furthermore, with a 21-bit packed encoding like UTF-21, not only byte-level endianness is relevant, but inner-byte bit-level endianness becomes relevant as well. While the encoding examples given imply a particular endianness, this is something that would have to be specified more explicitly.

chrismorgan 3y ago
Endianness is such a pain. I wish everything had always stayed big-endian, the generally superior option (though I will acknowledge it’s not quite uniformly superior). Implement any bit-level algorithms and endianness tends to rear its ugly head. I implemented the Speck cipher last year, and it basically mixed little- and big-endianness (byte/bit expression), but the paper never mentioned this at all, so that trial and error on the sample vectors, or emailing the authors for clarification, was the only way you’d find that out! The article gives examples that assume big-endianness, but in practice, UTF-16 is almost always UTF-16LE (no idea about UTF-32 since it’s not used so much), so the bytes within each code unit are, for practical purposes, back to front.

WorldMaker 3y ago

From the article: > I think UTF-32 is the simplest. Each number is put into a 32-bit integer, or 4 bytes. This is called a “fixed-width” encoding. This is wrong. The 4-byte fixed encoding is UCS-4, and while UCS-4 currently encodes all of Unicode, there's no guarantee it will "forever". Just as people who assumed UCS-2 was "all" of Unicode have since been proven (very) wrong. UTF-32 is a variable width encoding even if right now there's no reason to use surrogates out to the next plane (particularly because the next plane is entirely inaccessible to UTF-16 and only UTF-16, but UTF-16 is still one of the world's most common encodings).

e4m2 3y ago
UTF-32 is a fixed-width encoding. To quote directly from the Unicode standard, chapter 3.9, definition 90: > UTF-32 encoding form: The Unicode encoding form that assigns each Unicode scalar value to a single unsigned 32-bit code unit with the same numeric value as the Unicode scalar value. UCS-4 is defined in ISO 10646. Before the 2011 revision of the standard UCS-4 had a 31-bit codespace, since then the definitions of the ISO and Unicode standards have converged and UCS-4 and UTF-32 are now synonymous. > while UCS-4 currently encodes all of Unicode, there's no guarantee it will "forever". There is. Unicode has a codespace of U+0000..U+10FFFF. There are 2^16 + 2^20 − 2^11 assignable codepoints, in Unicode 15 a total of 149,186 (13.4%) have been assigned. Even if the Unicode codespace were to ever be extended again (it won't), the only encoding that would become incompatible is UTF-16. In fact, both UTF-8 and UTF-32 are trivially extensible and used to be wider encodings, but were restricted to 0x10FFFF arbitrarily to match UTF-16 limitations. EDIT: My wording is very ambiguous. There are actually 286,719 (25.8%) assigned codepoints in Unicode 15: https://www.unicode.org/versions/stats/charcountv15_0.html https://www.unicode.org/versions/stats/charcountv15_0.html. Still, the point stands.
- chrismorgan 3y ago
  > Even if the Unicode codespace were to ever be extended again (it won't), the only encoding that would become incompatible is UTF-16. In fact, both UTF-8 and UTF-32 are trivially extensible and used to be wider encodings, but were restricted to 0x10FFFF arbitrarily to match UTF-16 limitations. Mind you, it wouldn’t be this easy, because things should perform Unicode validation, and many do, so every piece of software would need to be updated for the new, enlarged version of Unicode, UTF-8 and UTF-32, and old software that validated would baulk or convert anything from the new ranges into REPLACEMENT CHARACTER. But yeah, UTF-16 would be toast.

OscarCunningham 3y ago

Would Huffman coding be called UTF-1?

erichocean 3y ago

The Factor language implemented Unicode support this way.