15 ms·

UTF-7: a ghost from the time before UTF-8

149 points by luu 8y ago

wglb 8y ago
It is unfortunate that they left off rfc4042: https://tools.ietf.org/html/rfc4042 https://tools.ietf.org/html/rfc4042, particularly since we are running out of room in bytes to stuff more bits.
- deleted 8y ago
  [deleted]
- masklinn 8y ago
  > we are running out of room in bytes to stuff more bits. The unicode codespace is composed of 17 16-bit planes, and so has room for 1114112 codepoints. As of Unicode 11, 137439 are allocated. And while UTF-8 has been restricted to only support 21 bits (which already allows almost 1 million more USV than actually allowed), the encoding scheme (and pre-2003 UTF-8) supports 31 bits of payload.
  - gpvos 8y ago
    Note the date of the RFC.
    - wglb 8y ago
      Of all of the ones on that date, this is my favorite.
- gumby 8y ago
  Apart from the date I think many modern programmers aren't aware of the influential PDP-10 (Mark was a PDP-10 programmer, which is how we met in the early 80s). The PDP-10 had 36-bit words which were byte addressable -- and bytes could range from 1-63 bits in length. So beyond the humorous language (nonets, septets) and inline PDP-10 assembly examples this was a absurdist joke intended to appeal on several levels.
fredley 8y ago
Oh God I was introduced to this parsing attachment names from raw SMTP emails. You think it's all fine until suddenly you come across a UTF-7-encoded UTF-8 filename...
- deleted 8y ago
  [deleted]
jperras 8y ago
Many years ago, UTF-7 was a viable way in which you could try to exploit Internet Explorer's somewhat questionable default of interpreting a resource as UTF-7 if it finds a UTF-7 character in the first 4096 bytes: http://shiflett.org/blog/2005/googles-xss-vulnerability http://shiflett.org/blog/2005/googles-xss-vulnerability
badrabbit 8y ago
Working with email,this such a painful thing to deal with. The IMAP legacy is all over outlook and email security/middleware appliances.
Sir_Cmpwn 8y ago
UTF-7 is the worst, but far from the only one. When writing an email client you will have to deal with no fewer than SIX distinct text encodings, including several unique to IMAP.
- michaelhoffman 8y ago
  Worse than UTF-EBCDIC?
Aardwolf 8y ago
I wish ASCII had a few more symbols and didn't waste 32 values on control codes. Missing characters imho: fixed width space, degree symbol, copyright/trademark symbols, the opposite direction of `, maybe a few more like pilcrow, section symbol, dagger, generic currency symbol, card suits, arrows and a few mathematical ones like +-, roughly equal, not equal, ... Why I care about ASCII here? Because programming language source code is still written in that, and it's also those symbols that appear on standard US keyboard layout. I think those extra symbols would have made life easier in many cases, more than some of the more obscure control characters it has (such as the useless line break vs newline we are still suffering from). They could have gotten away with just 16 instead of 32 control characters imho, even in the times when they had mechanical machines with bells :p
- vidanay 8y ago
  Control characters are still extremely common in embedded codes such as DataMatrix or QR Code. https://www.gs1.org/standards/barcodes https://www.gs1.org/standards/barcodes I would extrapolate that RFID uses similar encoding, but I have no experience there.
  - dfox 8y ago
    Code128 has special control symbols that are distinct from ASCII control characters which are used for framing/formating in context of GS1. Datamatrix also has special symbol that is equivalent to FNC1. The fact that scanners often encode FNC1 as some arbitrary ASCII character, with GS and ^ being common choices or even reformat the data to something resembling the human readable representation (ie. "(01)..." instead of FNC1 0 1 ...) is only about the scanner interface. But IIRC QR code does not have any special out-of-ASCII control symbols and GS1 structures are encoded with ASCII GS instead of FNC1.
- jperras 8y ago
  > more than some of the more obscure control characters it has It's perhaps an oddity now, but these control characters were most definitely necessary when ASCII was formalized in the 1960s from teletype machines.
simias 8y ago
>Even though this is 2018, occasionally someone will try to claim in conversation with me that UTF-16 is better than UTF-8. I'm curious, what are their arguments? UTF-16 is pretty objectively bad at everything, it's not compatible with ascii and C-strings like UTF-8 and it's not a decent constant-width encoding for codepoints like UTF-32 (and even that is not necessarily useful anyway since you often care more about characters than codepoints). AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts.
- beatgammit 8y ago
  I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compelling arguments in favor of it (aside from it being the default on Windows).
  - jchw 8y ago
    Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.
  - danieldk 8y ago
    I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts. Taking some random zh Wikipedia page in UTF-8: curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c 78543 curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m 71003 So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16. (Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)
devy 8y ago
Along the same lines of Unicode horror stories, my personal favorite is the MySQL's original 3-byte non-standard unicode implementation, called "utf8" and then later renamed to "utf8mb3". [1] It only covers the Basic Multilingual Plane (BMP). And it felt like the designed decision were made similar like those on MyISAM db engine that in retrospective, those were the wrong optimization that caused more trouble than bring benefits. And they didn't fix it until MySQL version 5.5.3 via introduction of "utf8mb4" charset. In a very obvious way, if you use emojis in your application with MySQL backend and not using the full 4-byte "utf8mb4" charset, you will absolutely get bitten by that gotcha. [2][3][4] [1]: https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8... [2]: https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434 https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u... [3]: https://stackoverflow.com/questions/202205/how-to-make-mysql-handle-utf-8-properly https://stackoverflow.com/questions/202205/how-to-make-mysql... [4]: https://www.eversql.com/mysql-utf8-vs-utf8mb4-whats-the-difference-between-utf8-and-utf8mb4/ https://www.eversql.com/mysql-utf8-vs-utf8mb4-whats-the-diff...
- tracker1 8y ago
  There should be a big FAT warning in the mysql/mariadb docs that "utf8" should NEVER be used, and "utf8mb4" is likely the intended use... A future full version cut should probably change the alias for "utf8" to the proper implementation. Came across this the last time I used mysql, fortunately before it was too far into the project. In any case, every time I've ever used mysql, there's something that pisses me off about it. Like indexing on a BINARY field is using case-insensitive comparisons, like it's actually text.
- dvtrn 8y ago
  if you use emojis in your application with MySQL backend and not using the full 4-byte "utf8mb4" charset, you will absolutely get bitten by that gotcha. Did we use to work together? Because I've been one of those poor DBAs to inherit a system that was not using utf8mb4 charsets, and sure damn enough an emoji took down said system for hours affecting every client on the books. They recovered, but limped along for another two years before eventually getting bought out and effectively soft-shutting down. I was hired post-Armageddon to keep the lights on and eventually left once the acquisition completed.
perlgeek 8y ago
Ok, I have a fun little UTF-7 story to share. At $work, we run a heavily patched OTRS for keeping track of our tickets, and have lots of systems automatically sending cron and other status mails to it (bad, I know). We got a bug report that the recipient email was displayed as Mojibake, something like blabla+⻧⻯⻱@ourdomain.net After digging into the source email and the OTRS code base, I found the problem: Some shitty MUA years ago failed to properly encode email headers, and sent 8-bit "Subject: " headers. To deal with that, OTRS had a workaround that tried to decode all (!) headers with the encoding specified in the Content-Encoding header, with a fallback to ASCII or Latin-1 if the decoding failed. Now, a pretty old Windows system or application sent UTF-7 encoded emails to something like blabla+autoreply=no+more=stuff-here@ourdomain.net, and OTRS successfully decoded the 'To:' header as UTF-7. And since it's not ASCII compatible, it turned it into gibberish. The "fix" was to only do the header decoding if the declared encoding is ASCII compatible. The code still seems to be in OTRS today: https://github.com/OTRS/otrs/blob/rel-6_0_14/Kernel/System/Encode.pm#L353 https://github.com/OTRS/otrs/blob/rel-6_0_14/Kernel/System/E...
amaccuish 8y ago
I'm looking forward to JMAP for the future of PIM.
rspeer 8y ago
UTF-7 is "fun" because encoding libraries tend to support it, but since nobody cares about it, edge cases in the implementation may go undiscovered for a while. Back on Python 2.7.5, the UTF-7 decoder didn't do range checking, so this script [1] produced a "Unicode string" containing the codepoint U+DEADBEEF. (The maximum valid codepoint is U+10FFFF.) This string would crash regexes, corrupt databases, etc., so that allowed denial-of-service attacks against any function that let you specify an arbitrary encoding. (This is fixed in all extant versions of Python.) [1] https://gist.github.com/rspeer/7559750 https://gist.github.com/rspeer/7559750
maxxxxx 8y ago
Is there anybody who really understands all the different encodings and character sets? I always get a headache when I have to analyze a problem in that area.
codezero 8y ago
For another "Unicode to ASCII" encoding, check out Punycode: https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode
Dylan16807 8y ago
> I looked inside three UTF-7 encoders and found they don't follow the RFC at all on this. Instead, they encode the UTF-16 to modified base64 without any zero bit padding, and then remove any base64 = padding from the result. That sounds like padding to a "character boundary" to me. I can't find anywhere that defines the term as being an entire block of 3/4.
tracker1 8y ago
Unless you're doing old-school serial/modem communication, I can't imagine anyone wanting to use UTF-7.

149 points by luu 8y ago

wglb 8y ago

It is unfortunate that they left off rfc4042: https://tools.ietf.org/html/rfc4042 https://tools.ietf.org/html/rfc4042, particularly since we are running out of room in bytes to stuff more bits.

deleted 8y ago
[deleted]
masklinn 8y ago
> we are running out of room in bytes to stuff more bits. The unicode codespace is composed of 17 16-bit planes, and so has room for 1114112 codepoints. As of Unicode 11, 137439 are allocated. And while UTF-8 has been restricted to only support 21 bits (which already allows almost 1 million more USV than actually allowed), the encoding scheme (and pre-2003 UTF-8) supports 31 bits of payload.
- gpvos 8y ago
  Note the date of the RFC.
  - wglb 8y ago
    Of all of the ones on that date, this is my favorite.
gumby 8y ago
Apart from the date I think many modern programmers aren't aware of the influential PDP-10 (Mark was a PDP-10 programmer, which is how we met in the early 80s). The PDP-10 had 36-bit words which were byte addressable -- and bytes could range from 1-63 bits in length. So beyond the humorous language (nonets, septets) and inline PDP-10 assembly examples this was a absurdist joke intended to appeal on several levels.

fredley 8y ago

Oh God I was introduced to this parsing attachment names from raw SMTP emails. You think it's all fine until suddenly you come across a UTF-7-encoded UTF-8 filename...

deleted 8y ago
[deleted]

jperras 8y ago

Many years ago, UTF-7 was a viable way in which you could try to exploit Internet Explorer's somewhat questionable default of interpreting a resource as UTF-7 if it finds a UTF-7 character in the first 4096 bytes: http://shiflett.org/blog/2005/googles-xss-vulnerability http://shiflett.org/blog/2005/googles-xss-vulnerability

badrabbit 8y ago

Working with email,this such a painful thing to deal with. The IMAP legacy is all over outlook and email security/middleware appliances.

Sir_Cmpwn 8y ago

UTF-7 is the worst, but far from the only one. When writing an email client you will have to deal with no fewer than SIX distinct text encodings, including several unique to IMAP.

michaelhoffman 8y ago
Worse than UTF-EBCDIC?

Aardwolf 8y ago

I wish ASCII had a few more symbols and didn't waste 32 values on control codes. Missing characters imho: fixed width space, degree symbol, copyright/trademark symbols, the opposite direction of `, maybe a few more like pilcrow, section symbol, dagger, generic currency symbol, card suits, arrows and a few mathematical ones like +-, roughly equal, not equal, ... Why I care about ASCII here? Because programming language source code is still written in that, and it's also those symbols that appear on standard US keyboard layout. I think those extra symbols would have made life easier in many cases, more than some of the more obscure control characters it has (such as the useless line break vs newline we are still suffering from). They could have gotten away with just 16 instead of 32 control characters imho, even in the times when they had mechanical machines with bells :p

vidanay 8y ago
Control characters are still extremely common in embedded codes such as DataMatrix or QR Code. https://www.gs1.org/standards/barcodes https://www.gs1.org/standards/barcodes I would extrapolate that RFID uses similar encoding, but I have no experience there.
- dfox 8y ago
  Code128 has special control symbols that are distinct from ASCII control characters which are used for framing/formating in context of GS1. Datamatrix also has special symbol that is equivalent to FNC1. The fact that scanners often encode FNC1 as some arbitrary ASCII character, with GS and ^ being common choices or even reformat the data to something resembling the human readable representation (ie. "(01)..." instead of FNC1 0 1 ...) is only about the scanner interface. But IIRC QR code does not have any special out-of-ASCII control symbols and GS1 structures are encoded with ASCII GS instead of FNC1.
jperras 8y ago
> more than some of the more obscure control characters it has It's perhaps an oddity now, but these control characters were most definitely necessary when ASCII was formalized in the 1960s from teletype machines.

simias 8y ago

>Even though this is 2018, occasionally someone will try to claim in conversation with me that UTF-16 is better than UTF-8. I'm curious, what are their arguments? UTF-16 is pretty objectively bad at everything, it's not compatible with ascii and C-strings like UTF-8 and it's not a decent constant-width encoding for codepoints like UTF-32 (and even that is not necessarily useful anyway since you often care more about characters than codepoints). AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts.

beatgammit 8y ago
I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compelling arguments in favor of it (aside from it being the default on Windows).
- jchw 8y ago
  Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.
- danieldk 8y ago
  I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts. Taking some random zh Wikipedia page in UTF-8: curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c 78543 curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m 71003 So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16. (Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)

devy 8y ago

Along the same lines of Unicode horror stories, my personal favorite is the MySQL's original 3-byte non-standard unicode implementation, called "utf8" and then later renamed to "utf8mb3". [1] It only covers the Basic Multilingual Plane (BMP). And it felt like the designed decision were made similar like those on MyISAM db engine that in retrospective, those were the wrong optimization that caused more trouble than bring benefits. And they didn't fix it until MySQL version 5.5.3 via introduction of "utf8mb4" charset. In a very obvious way, if you use emojis in your application with MySQL backend and not using the full 4-byte "utf8mb4" charset, you will absolutely get bitten by that gotcha. [2][3][4] [1]: https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8... [2]: https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434 https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-u... [3]: https://stackoverflow.com/questions/202205/how-to-make-mysql-handle-utf-8-properly https://stackoverflow.com/questions/202205/how-to-make-mysql... [4]: https://www.eversql.com/mysql-utf8-vs-utf8mb4-whats-the-difference-between-utf8-and-utf8mb4/ https://www.eversql.com/mysql-utf8-vs-utf8mb4-whats-the-diff...

tracker1 8y ago
There should be a big FAT warning in the mysql/mariadb docs that "utf8" should NEVER be used, and "utf8mb4" is likely the intended use... A future full version cut should probably change the alias for "utf8" to the proper implementation. Came across this the last time I used mysql, fortunately before it was too far into the project. In any case, every time I've ever used mysql, there's something that pisses me off about it. Like indexing on a BINARY field is using case-insensitive comparisons, like it's actually text.
dvtrn 8y ago
if you use emojis in your application with MySQL backend and not using the full 4-byte "utf8mb4" charset, you will absolutely get bitten by that gotcha. Did we use to work together? Because I've been one of those poor DBAs to inherit a system that was not using utf8mb4 charsets, and sure damn enough an emoji took down said system for hours affecting every client on the books. They recovered, but limped along for another two years before eventually getting bought out and effectively soft-shutting down. I was hired post-Armageddon to keep the lights on and eventually left once the acquisition completed.

perlgeek 8y ago

Ok, I have a fun little UTF-7 story to share. At $work, we run a heavily patched OTRS for keeping track of our tickets, and have lots of systems automatically sending cron and other status mails to it (bad, I know). We got a bug report that the recipient email was displayed as Mojibake, something like blabla+⻧⻯⻱@ourdomain.net After digging into the source email and the OTRS code base, I found the problem: Some shitty MUA years ago failed to properly encode email headers, and sent 8-bit "Subject: " headers. To deal with that, OTRS had a workaround that tried to decode all (!) headers with the encoding specified in the Content-Encoding header, with a fallback to ASCII or Latin-1 if the decoding failed. Now, a pretty old Windows system or application sent UTF-7 encoded emails to something like blabla+autoreply=no+more=stuff-here@ourdomain.net, and OTRS successfully decoded the 'To:' header as UTF-7. And since it's not ASCII compatible, it turned it into gibberish. The "fix" was to only do the header decoding if the declared encoding is ASCII compatible. The code still seems to be in OTRS today: https://github.com/OTRS/otrs/blob/rel-6_0_14/Kernel/System/Encode.pm#L353 https://github.com/OTRS/otrs/blob/rel-6_0_14/Kernel/System/E...

amaccuish 8y ago

I'm looking forward to JMAP for the future of PIM.

rspeer 8y ago

UTF-7 is "fun" because encoding libraries tend to support it, but since nobody cares about it, edge cases in the implementation may go undiscovered for a while. Back on Python 2.7.5, the UTF-7 decoder didn't do range checking, so this script [1] produced a "Unicode string" containing the codepoint U+DEADBEEF. (The maximum valid codepoint is U+10FFFF.) This string would crash regexes, corrupt databases, etc., so that allowed denial-of-service attacks against any function that let you specify an arbitrary encoding. (This is fixed in all extant versions of Python.) [1] https://gist.github.com/rspeer/7559750 https://gist.github.com/rspeer/7559750

maxxxxx 8y ago

Is there anybody who really understands all the different encodings and character sets? I always get a headache when I have to analyze a problem in that area.

codezero 8y ago

For another "Unicode to ASCII" encoding, check out Punycode: https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode

Dylan16807 8y ago

> I looked inside three UTF-7 encoders and found they don't follow the RFC at all on this. Instead, they encode the UTF-16 to modified base64 without any zero bit padding, and then remove any base64 = padding from the result. That sounds like padding to a "character boundary" to me. I can't find anywhere that defines the term as being an entire block of 3/4.

tracker1 8y ago

Unless you're doing old-school serial/modem communication, I can't imagine anyone wanting to use UTF-7.