4 ms·

> Unicode has three official ways of storing these numbers: UTF-8, UTF-16, and UTF-32 They are less official, but Unicode defines several other encodings in th

> Unicode has three official ways of storing these numbers: UTF-8, UTF-16, and UTF-32 They are less official, but Unicode defines several other encodings in their technical reports and notes: SCSU: https://www.unicode.org/reports/tr6/tr6-4.html https://www.unicode.org/reports/tr6/tr6-4.html UTF-EBCDIC: http://www.unicode.org/reports/tr16/tr16-8.html http://www.unicode.org/reports/tr16/tr16-8.html CESU-8: https://www.unicode.org/reports/tr26/tr26-4.html https://www.unicode.org/reports/tr26/tr26-4.html BOCU-1: https://www.unicode.org/notes/tn6/ https://www.unicode.org/notes/tn6/ These are all somewhat legacy, having never seen very wide adoption: Compression schemes such as SCSU and BOCU-1 turned out in practice to add a lot of complexity for minimal real-world benefit. If you really need to save storage, then a standard compression algorithm (e.g. deflate or zstd) is usually a better option. CESU-8 is a backward compatibility hack for old UTF-8 implementations which used encoded UTF-16 surrogates for codepoints outside the BMP. UTF-EBCDIC is basically UTF-8 but for EBCDIC. It was invented by someone at IBM, but IBM themselves ended up standardising on UTF-16 instead. And if IBM ended up not using UTF-EBCDIC, who else was going to? Well, there's a few surviving non-IBM mainframe vendors who also use EBCDIC (such as Unisys and Fujitsu BS2000), but I'm not aware any of them ever expressed any interest in it either.

wpietri 3y ago
> UTF-EBCDIC Just reading that made my eyelid twitch.
- ElectricalUnion 3y ago
  Almost nothing uses UTF-EBCDIC by default, db2 (and the underlying VSAM persistent storage) mostly ends up writing UTF-16 (but you can set it up to write UTF-8); except for internal catalog data and network traffic, that are usually UTF-8 no matter what.
  - skissane 3y ago
    Do any IBM products have UTF-EBCDIC as a supported configuration option? Does anyone use it? Oracle RDBMS supports "UTFE" on EBCDIC platforms, which is to UTF-EBCDIC what CESU-8 is to UTF-8 – i.e. UTF-EBCDIC but with codepoints outside the BMP encoded using surrogates. Over the last couple of decades, the only EBCDIC platforms Oracle RDBMS has been supported on have been z/OS and BS2000/OSD – once upon a time it was also available on VM/CMS (not sure when that was discontinued), and possibly more besides (in the 80s thru early 90s, Oracle ported their DB to just about everything under the sun, they later became much more selective in what they'd support). The z/OS port was discontinued with version 10gR2 (initial release 2005, final patchset 2010) – although, while Oracle normally puts a time limit on their pay-extra support for patches for old versions ("extended support"), exceptionally they've said for the z/OS port they'll continue offering that as long as customers want it. So the only remaining EBCDIC port of Oracle RDBMS is Fujitsu BS2000/OSD – I think the only reason that port survives is because Fujitsu pays Oracle to keep on producing it. mksh (MirBSD's ksh) supports some custom EBCDIC encoding of UTF-8 (so-called "nega-UTF-8"), instead of proper UTF-EBCDIC. One difference is that UTF-EBCDIC avoids using the C1 controls (bytes 0x80 thru 0x9F), so EBCDIC control characters can be left intact, whereas "nega-UTF-8" just converts UTF-8 to EBCDIC using an arbitrary EBCDIC code page, hence failing to fully preserve the full complement of EBCDIC control characters. Also, while UTF-EBCDIC is based on a fixed EBCDIC code page (1047), "nega-UTF-8" isn't, so it really isn't a single encoding, rather a family of them. Has anyone else ever implemented UTF-EBCDIC, or something similar (given neither Oracle UTFE nor mksh's nega-UTF-8 are actually UTF-EBCDIC)? Years ago, I wrote an implementation of it myself (in Java), which I've never released – not for any work-related purpose, just as a private exercise in recreational programming. Actually, I have an FTP server I wrote which serves up UTF-EBCDIC as "TYPE E", but I could never test it properly, because (at the time) I couldn't find an FTP client which actually implemented TYPE E. I think I later found one, but by then I'd lost interest in pursuing it any further.
mort96 3y ago
UTF-8 with surrogate pairs has to be the worst idea I've heard yet this month.
kps 3y ago
Don't forget RFC4042, which defines UTF-9 and UTF-18.
- skissane 3y ago
  Sadly, both UTF-9 and UTF-18 are poorly designed. If you look at the structure of UTF-8, it is based on a prefix code: each byte starts with a string of 1s followed by a 0, where the total number of bytes in the sequence is the number of initial 1 bits in the first byte – except for single-byte sequences, which start with a 0 bit, and the 10 prefix instead of being used to introduce single byte sequences is instead used to mark continuation bytes. There is no reason why UTF-9 could not have adopted the same structure – 0xxxxxxxx for U+0000 thru U+00FF, 110xxxxxx 10yyyyyyy for U+0100 thr thru U+1FFF, etc, but instead the UTF-9 author decided to go with a much more simplistic scheme of "initial bit is 0 for initial byte, 1 for continuation byte". UTF-18 is also flawed, because it has no concept of surrogates, hence it cannot represent the full 21-bit repertoire of Unicode characters. Once again, there is no technical reason why it could not have supported the use of surrogates to achieve this. I suppose it doesn't really matter because it was just an April Fool's joke not a serious proposal – but if one was to make a serious proposal for a UTF-9 and UTF-18, I think it would have to fix these two flaws in RFC4042.

wpietri 3y ago

> UTF-EBCDIC Just reading that made my eyelid twitch.

ElectricalUnion 3y ago
Almost nothing uses UTF-EBCDIC by default, db2 (and the underlying VSAM persistent storage) mostly ends up writing UTF-16 (but you can set it up to write UTF-8); except for internal catalog data and network traffic, that are usually UTF-8 no matter what.
- skissane 3y ago
  Do any IBM products have UTF-EBCDIC as a supported configuration option? Does anyone use it? Oracle RDBMS supports "UTFE" on EBCDIC platforms, which is to UTF-EBCDIC what CESU-8 is to UTF-8 – i.e. UTF-EBCDIC but with codepoints outside the BMP encoded using surrogates. Over the last couple of decades, the only EBCDIC platforms Oracle RDBMS has been supported on have been z/OS and BS2000/OSD – once upon a time it was also available on VM/CMS (not sure when that was discontinued), and possibly more besides (in the 80s thru early 90s, Oracle ported their DB to just about everything under the sun, they later became much more selective in what they'd support). The z/OS port was discontinued with version 10gR2 (initial release 2005, final patchset 2010) – although, while Oracle normally puts a time limit on their pay-extra support for patches for old versions ("extended support"), exceptionally they've said for the z/OS port they'll continue offering that as long as customers want it. So the only remaining EBCDIC port of Oracle RDBMS is Fujitsu BS2000/OSD – I think the only reason that port survives is because Fujitsu pays Oracle to keep on producing it. mksh (MirBSD's ksh) supports some custom EBCDIC encoding of UTF-8 (so-called "nega-UTF-8"), instead of proper UTF-EBCDIC. One difference is that UTF-EBCDIC avoids using the C1 controls (bytes 0x80 thru 0x9F), so EBCDIC control characters can be left intact, whereas "nega-UTF-8" just converts UTF-8 to EBCDIC using an arbitrary EBCDIC code page, hence failing to fully preserve the full complement of EBCDIC control characters. Also, while UTF-EBCDIC is based on a fixed EBCDIC code page (1047), "nega-UTF-8" isn't, so it really isn't a single encoding, rather a family of them. Has anyone else ever implemented UTF-EBCDIC, or something similar (given neither Oracle UTFE nor mksh's nega-UTF-8 are actually UTF-EBCDIC)? Years ago, I wrote an implementation of it myself (in Java), which I've never released – not for any work-related purpose, just as a private exercise in recreational programming. Actually, I have an FTP server I wrote which serves up UTF-EBCDIC as "TYPE E", but I could never test it properly, because (at the time) I couldn't find an FTP client which actually implemented TYPE E. I think I later found one, but by then I'd lost interest in pursuing it any further.

mort96 3y ago

UTF-8 with surrogate pairs has to be the worst idea I've heard yet this month.

kps 3y ago

Don't forget RFC4042, which defines UTF-9 and UTF-18.

skissane 3y ago
Sadly, both UTF-9 and UTF-18 are poorly designed. If you look at the structure of UTF-8, it is based on a prefix code: each byte starts with a string of 1s followed by a 0, where the total number of bytes in the sequence is the number of initial 1 bits in the first byte – except for single-byte sequences, which start with a 0 bit, and the 10 prefix instead of being used to introduce single byte sequences is instead used to mark continuation bytes. There is no reason why UTF-9 could not have adopted the same structure – 0xxxxxxxx for U+0000 thru U+00FF, 110xxxxxx 10yyyyyyy for U+0100 thr thru U+1FFF, etc, but instead the UTF-9 author decided to go with a much more simplistic scheme of "initial bit is 0 for initial byte, 1 for continuation byte". UTF-18 is also flawed, because it has no concept of surrogates, hence it cannot represent the full 21-bit repertoire of Unicode characters. Once again, there is no technical reason why it could not have supported the use of surrogates to achieve this. I suppose it doesn't really matter because it was just an April Fool's joke not a serious proposal – but if one was to make a serious proposal for a UTF-9 and UTF-18, I think it would have to fix these two flaws in RFC4042.