25 ms·

The history of UTF-8 as told by Rob Pike (2003)

244 points by quyleanh 6y ago

mids_hn 6y ago
> the ability to synchronize a byte stream picked up mid-run, with less that one character being consumed before synchronization Can somebody explain or link to an explanation on how UTF-8 allows for this?
- legulere 6y ago
  All trailing bytes and only trailing bytes are of the form 10xxxxxx. If you read such a byte you just have to iterate backwards until you find a non-trailing byte.
- saagarjha 6y ago
  It’s fairly simple, actually: leading bytes have a specific bit pattern that continuation bytes don’t. A single-byte character will have the topmost bit unset (0b0xxxxxx), and for a multi-byte run the first byte will have the top two bits set (0b11xxxxxx) and any succeeding bytes will have the top bit set but the next bit unset (0b10xxxxxx). This means given an arbitrary byte you can always tell what it is, and you can tell when you’re at the start of a next character by looking for those first two bit patterns.
- mjevans 6y ago
  The upper bits of the FIRST octet are used to determine the run length of the sequence. All of the other bytes in the sequence use the upper two bits (0xC prefix len 2 OR b10xxxxxx) to indicate that it's another 6 bits of data for the current character. If synchronization is lost mid-character, by definition that interrupted character is lost. However the very next complete character will be clearly indicated by a byte beginning with either no sign (a 7 bit character) OR a number of 1s indicating the octet count followed by a zero. This is covered in the section titled: Proposed FSS-UTF ---------------- ... Bits Hex Min Hex Max Byte Sequence in Binary 1 7 00000000 0000007f 0vvvvvvv 2 11 00000080 000007FF 110vvvvv 10vvvvvv 3 16 00000800 0000FFFF 1110vvvv 10vvvvvv 10vvvvvv ... Examples trimmed for mobile.
- louai 6y ago
  Note that it says less than one character. A character in UTF-8 can be composed of multiple bytes. The encoding scheme is laid out in the linked email. Based on the high bits it's possible to detect when a new character starts. Relevant portion: We define 7 byte types: T0 0xxxxxxx 7 free bits Tx 10xxxxxx 6 free bits T1 110xxxxx 5 free bits T2 1110xxxx 4 free bits T3 11110xxx 3 free bits T4 111110xx 2 free bits T5 111111xx 2 free bits Encoding is as follows. >From hex Thru hex Sequence Bits 00000000 0000007f T0 7 00000080 000007FF T1 Tx 11 00000800 0000FFFF T2 Tx Tx 16 00010000 001FFFFF T3 Tx Tx Tx 21 00200000 03FFFFFF T4 Tx Tx Tx Tx 26 04000000 FFFFFFFF T5 Tx Tx Tx Tx Tx 32 [...] 4. All of the sequences synchronize on any byte that is not a Tx byte. If you are starting mid-run, skip initial Tx bytes. That will always be less than one character.
  - ChrisSD 6y ago
    Note that UTF-8 has since been restricted to at most 4 bytes (i.e. the longest sequence is `T3 Tx Tx Tx`).
    - throwdbaaway 6y ago
      So now we know who is really responsible for the whole MySQL utf8mb4 fiasco -- these 2 guys sitting in a diner, conjuring up a brilliant scheme to cover 4 billions characters, which turned out to exceed the actual requirement by more than 2000x. September 1992: 2 guys scribbling on a placemat. January 1998: RFC 2279 defines UTF-8 to be between 1 to 6 bytes. March 2001: A bunch of CJK characters were added to Unicode Data 3.1.0, pushing the total to 94,140, exceeding the 16-bit limit of 3 bytes UTF-8. March 2002: MySQL added support for UTF-8, initially setting the limit to 6 bytes (https://github.com/mysql/mysql-server/commit/55e0a9c https://github.com/mysql/mysql-server/commit/55e0a9c) September 2002: MySQL decided to reduce the limit to 3 bytes, probably for storage efficiency reason (https://github.com/mysql/mysql-server/commit/43a506c https://github.com/mysql/mysql-server/commit/43a506c, https://adamhooper.medium.com/in-mysql-never-use-utf8-use-utf8mb4-11761243e434 https://adamhooper.medium.com/in-mysql-never-use-utf8-use-ut...) November 2003: RFC 3629 defines UTF-8 to be between 1 to 4 bytes. Arguably, if the placemat was smaller and the guys stopped at 4 bytes after running out of space, perhaps MySQL would have done the right thing? Ah, who am I kidding. The same commit would likely still happen. EDIT: Just notice this in the footnotes, and the plot thickens... > The 4, 5, and 6 byte sequences are only there for political reasons. I would prefer to delete these. So UTF-8 was indeed intended to be utf8mb3!
- ChrisSD 6y ago
  Other people have answered your question but I wanted to clarify one point. The word "character" here means "unicode code point". However, what the user thinks of as a single character can be made up of more than one code point. This presents a different problem and one UTF-8 itself can't help with. The Unicode Consortium has a report on extended grapheme clusters[0] (i.e. user-perceived characters). Essentially, if you're processing some text mid stream, it might not be clear if a code point is the start of a new user-perceived character or not. So you may want to skip ahead until an unambiguous symbol boundary is reached. [0]: https://www.unicode.org/reports/tr29/ https://www.unicode.org/reports/tr29/
- fomine3 6y ago
  It is called Self-synchronizing code, simple and beautiful design. https://en.wikipedia.org/wiki/Self-synchronizing_code https://en.wikipedia.org/wiki/Self-synchronizing_code
VMG 6y ago
https://web.archive.org/web/20210324065459/http://doc.cat-v.org/bell_labs/utf-8_history https://web.archive.org/web/20210324065459/http://doc.cat-v....
janvdberg 6y ago
More or less same the story but in an explainer video: 'Characters, Symbols and the Unicode Miracle - Computerphile' https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4
- not_knuth 6y ago
  This video is fantastic, especially if you've never really understood how Unicode worked before. Keeps it simple and gets to the point. I remember seeing this a couple of years ago and thinking "If only this existed when I learned about UTF-8, this would've saved me a lot of bad explanations and time". This is now my goto-video if anyone asks me how Unicode works.
  - enriquto 6y ago
    > This video is fantastic, especially if you've never really understood how Unicode worked before. Can you people really learn things on videos? My brain sort of shuts down on audiovisual material, I can only really watch and understand light movies. For more complicated material, I can only learn it by reading. There's so much essential back and forth that is impossible on a video. With a text, you have everything already in front of you and your eyes and mind can wander freely. Maybe it's just me, but I really can't stand the fixed, inflexible rhythm that is imposed by listening to speech.
    - tux3 6y ago
      I also like text, but if you're genuinely asking, yes! Video evidently works great for a lot of people. You can't seek it as well as text, so instead rely on your memory more to remember things that are still unclear or don't make sense yet, and see when they are explained later. It's okay to have different learning styles I think, it's not too surprising to me that some prefer different medium
      - enriquto 6y ago
        It's precisely the seeking that kills me. The only interface for seeking a video is a tiny, one-dimensional bar. How do you remember exactly where the right information occurs? On the other hand, text is visible as a whole, and the "seeking" is two-dimensional. Much more efficient to seek. I have a good visual memory and I tend to remember exactly where on a page the formula that I want appears, and just glance away at it without moving any muscles other than my eyes. To seek a part on a video, the process is overwhelmingly more complicated. And then the place where I was before the seek is lost, and you have to seek for it again. Argh! Thanks for the answer, it turns out that we people are wildly different to each other.
TorKlingberg 6y ago
I think we should take a moment to appreciate how great UTF-8 is, and how it well it worked out. It's easy to get disillusioned with internet standards when IPv6 is taking forever and messaging is all proprietary locked down protocols. Yet character encodings used to be a horrible mess and now it's not. In the 90's the only practical solution was for everyone to use the same OS, same word processor, same web browser, and who cares about talking with foreigners anyway? I don't think it was always guaranteed to turn out well. China and Japan could have stayed with their own encodings. Microsoft and Apple could have done incompatible things. The tech world is full of bad things we're stuck with because there's no way to coordinate a change. Unicode has it's flaws, UTF-16 is still lurking here and there, everyone loves to argue about emoji, but overall text just works now.
- roblabla 6y ago
  > UTF-16 is still lurking here and there As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16/UCS2 as the string encoding. Also fun fact, on the Nintendo Switch, various subsystems use different kind of encoding. The filsystem submodule uses Shift-JIS, most of the other modules use UTF-8, but some others yet use UTF-16 (like the virtual keyboard, IIRC). A brilliant mess.
  - ChrisSD 6y ago
    Personally I don't find UTF-16 to be too bad. It's a simple encoding and very easy to convert to/from UTF-8. So your program can be written in UTF-8 and your WinAPI wrappers can convert as/when needed.
    - jeltz 6y ago
      The bad thing with UTF-16 is that so much software assumes that one code point always is 16 bits.
      - cygx 6y ago
        Most of the hard stuff is there no matter the encoding (normalization, user-perceived characters spanning multiple code units, paths vs strings, ...).
bartwe 6y ago
Was always a bit surprised that if you had the length encoding byte, that it didn't also imply an offset by the values that should have been encoded with a shorter sequence
- Tuna-Fish 6y ago
  This is there to make decoding faster. I think it's a mistake.
  - tacitusarc 6y ago
    The continuation bits also allow backwards traversal and proper null byte decoding.
0x0 6y ago
Interesting comment near the end, "2. The 4, 5, and 6 byte sequences are only there for political reasons. I would prefer to delete these." If that had happened, I guess emojis as we know them today might never have happened, since it would have limited us to 16 bits of code points. Or we would have had to start doing surrogate pairs even in UTF-8. Close call.
- morelisp 6y ago
  The 5-6 byte variants (and also 4 at the time) exist because of the need to round-trip UCS surrogate pairs through UTF-8, no? That's what I assume the "political reasons" are...
  - deleted 6y ago
    [deleted]
  - 0x0 6y ago
    I don't think so? Aren't UCS surrogate pairs at most 16bit each by their very purpose? Also, >16bit unicode code points came much later, I believe, in Unicode 2.0 in 1996 according to Wikipedia (vs UTF-8 which is from around 1992)
  - ChrisSD 6y ago
    Surrogate pairs are only in UTF-16 so as to encode code points that require more than 16 bits. UTF-8 has no need of them because it's already a variable width encoding. If there were no code points larger than 16 bits then UTF-8 would only need a maximum of 3 bytes per code point and UTF-16 wouldn't need surrogate pairs. Well actually UTF-16 probably wouldn't exist at all because UCS-2 would have been enough for everybody.
    - deleted 6y ago
      [deleted]
  - Sniffnoy 6y ago
    Others have already answered why surrogate pairs are irrelevant (and not UCS), but I think it's worth saying what the probable actual reason for 5-6 byte variants was. Remember that UCS and Unicode were at this point still two separate things; Unicode was supposed to be 16-bit (and later it got expanded, causing the whole surrogates mess), while UCS was supposed to be 31-bit. I assume the 5-6 byte variants were for UCS (back before it got merged with Unicode).
YesThatTom2 6y ago
ACM should put a plaque at that diner.
- euroderf 6y ago
  Yes but at what diner ? Wikipedia says NJ has about 525.
  - YesThatTom2 6y ago
    We could ask. I've always assumed Prestige Diner & Restaurant just because it is close to the Murray Hill building.
    - v64 6y ago
      Another commenter stated it's Corner Café in New Providence [1] [1] https://news.ycombinator.com/item?id=26739049 https://news.ycombinator.com/item?id=26739049
dpcx 6y ago
Designing and coding up something of this importance in less than a week is impressive. Strangely, what I'm more impressed with is that, in 2003 - 11 years after all of this happened - Bell Labs still had _sendmail logs for two users of their system_. How much history do we throw away by trashing logs nowadays? Sure, there's probably a lot more traffic... but wow.
- yiyus 6y ago
  Plan 9 always had great archival utilities. I do not know what they used at those early days, but I guess that at the point rsc recovered it, it was stored in the venti server: https://p9f.org/sys/doc/venti/venti.html https://p9f.org/sys/doc/venti/venti.html
deleted 6y ago
[deleted]
haolez 6y ago
One of Rob's ideas - the GOPATH - is something that I enjoy very much and even port the concept to other languages, sometimes. However, it seems that the community as a whole didn't embrace it. I feel isolated, but in good company :D
- BlueTemplar 6y ago
  Please tell us more?
- yobert 6y ago
  I liked GOPATH at first, but the limitations of projects stepping on each other quickly made me sad about it. Now that go modules has removed the need for GOPATH, I'm very happy about it being obsolete. :)
peterdepasquale 6y ago
Those of us from NJ really need to know what diner though... > UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992.
- kps 6y ago
  “The diner was the Corner Café in New Providence, New Jersey.” — Rob Pike, https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-years-old-in-2012.html https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-y...
BlueTemplar 6y ago
The transition from 32 bit to 64 bit words was a missed opportunity to move from 8 bit to 32 bit bytes, which would have greatly simplified the fundamental aspect that is dealing with text in computers.
- colejohnson66 6y ago
  Some architectures use 8 bit bytes, but not really. What I mean is: some architectures require a word bigger than 8 bits (16+) be aligned. As in: to read a 32 bit word from memory, the first octet must have an address ending in two zero bits. But if you want to read only a byte, alignment doesn’t matter. IIRC, ARM used to have this “alignment” requirement.
  - BlueTemplar 6y ago
    Why ? When ?
ChrisArchitect 6y ago
similar post of the same email/story, 15 days ago and a year ago here's more discussion from then: https://news.ycombinator.com/item?id=21212445 https://news.ycombinator.com/item?id=21212445
dang 6y ago
If curious, past threads: UTF-8 history (2003) - https://news.ycombinator.com/item?id=21212445 https://news.ycombinator.com/item?id=21212445 - Oct 2019 (52 comments) UTF-8 History (2003) - https://news.ycombinator.com/item?id=19565980 https://news.ycombinator.com/item?id=19565980 - April 2019 (3 comments) UTF-8 history - https://news.ycombinator.com/item?id=15236856 https://news.ycombinator.com/item?id=15236856 - Sept 2017 (1 comment) UTF-8 history (2003) - https://news.ycombinator.com/item?id=8648541 https://news.ycombinator.com/item?id=8648541 - Nov 2014 (7 comments) UTF-8 Original Proposal - https://news.ycombinator.com/item?id=6463466 https://news.ycombinator.com/item?id=6463466 - Sept 2013 (3 comments) UTF-8 History - https://news.ycombinator.com/item?id=2081932 https://news.ycombinator.com/item?id=2081932 - Jan 2011 (2 comments) The history of UTF-8 as told by Rob Pike - https://news.ycombinator.com/item?id=577116 https://news.ycombinator.com/item?id=577116 - April 2009 (1 comment) --- Don't miss this great link from the 2017 thread: https://www.flickr.com/photos/ajstarks/sets/72157631470798870 https://www.flickr.com/photos/ajstarks/sets/7215763147079887...
pierrebai 6y ago
One thing to realize is that UTF-8 is here and required only because we're all stuck with limitations based on the low-power, low-limitations of early systems. There is not a single reason, other than historical, backward compatibility and the overall ecosystem putting incommensurate pressure on all future systems, why the basic memory unit is 8 bits. We're stuck on 8 bits because all API, OS and specifications assume the basic units is 8 bits. If computer memory had evolved faster, or computer usage spread slower, the basic unit could have been 32 bits, or even 64 bits. Think how much easier things would be. UTF could be encoded as a single unit, without ambiguous order. What we've lost due to byte and UTF-8 is for example the text editor as the universal editor. Due to the need of UTF-8 decoding, we're no longer able to open a any (possibly binary) file as text and save it reliably after editing one character. There are other consequences, for example at the file-system level for file names, corruption, how to handle invalid UTF-8 sequences, etc. (And before you talk about the byte order problem, that is the point: there is no order problem for the basic unit you choose. No one (outside of some low-level 1-bit serial interfaces) cars about bit order or nibble order of bytes. Because the bit order is hidden within all interface accepting byte as the unit.)
- cesarb 6y ago
  It's hard to argue against a counterfactual, but I believe there is a "sweet spot" for the size of the basic memory unit, and that "sweet spot" is probably in the range from 4 to 16 bits (power of two sizes are desirable, so the possible sizes would be 4 bits, 8 bits, and 16 bits), so even if computers evolved from scratch today already starting with large memories, the basic unit of memory addressing still wouldn't be that big. My guess is that the chosen size would be 16 bits.
cryptonector 6y ago
> helix: Sep 8 03:22:13: ken: upas/sendmail: remote inet!xopen.co.uk!xojig > From ken Tue Sep 8 03:22:07 EDT 1992 (xojig@xopen.co.uk) 6833 It was 1992 and they were still using UUCP-style addressing.

244 points by quyleanh 6y ago

> the ability to synchronize a byte stream picked up mid-run, with less that one character being consumed before synchronization Can somebody explain or link to an explanation on how UTF-8 allows for this?

legulere 6y ago
All trailing bytes and only trailing bytes are of the form 10xxxxxx. If you read such a byte you just have to iterate backwards until you find a non-trailing byte.
saagarjha 6y ago
It’s fairly simple, actually: leading bytes have a specific bit pattern that continuation bytes don’t. A single-byte character will have the topmost bit unset (0b0xxxxxx), and for a multi-byte run the first byte will have the top two bits set (0b11xxxxxx) and any succeeding bytes will have the top bit set but the next bit unset (0b10xxxxxx). This means given an arbitrary byte you can always tell what it is, and you can tell when you’re at the start of a next character by looking for those first two bit patterns.
mjevans 6y ago
The upper bits of the FIRST octet are used to determine the run length of the sequence. All of the other bytes in the sequence use the upper two bits (0xC prefix len 2 OR b10xxxxxx) to indicate that it's another 6 bits of data for the current character. If synchronization is lost mid-character, by definition that interrupted character is lost. However the very next complete character will be clearly indicated by a byte beginning with either no sign (a 7 bit character) OR a number of 1s indicating the octet count followed by a zero. This is covered in the section titled: Proposed FSS-UTF ---------------- ... Bits Hex Min Hex Max Byte Sequence in Binary 1 7 00000000 0000007f 0vvvvvvv 2 11 00000080 000007FF 110vvvvv 10vvvvvv 3 16 00000800 0000FFFF 1110vvvv 10vvvvvv 10vvvvvv ... Examples trimmed for mobile.
louai 6y ago
Note that it says less than one character. A character in UTF-8 can be composed of multiple bytes. The encoding scheme is laid out in the linked email. Based on the high bits it's possible to detect when a new character starts. Relevant portion: We define 7 byte types: T0 0xxxxxxx 7 free bits Tx 10xxxxxx 6 free bits T1 110xxxxx 5 free bits T2 1110xxxx 4 free bits T3 11110xxx 3 free bits T4 111110xx 2 free bits T5 111111xx 2 free bits Encoding is as follows. >From hex Thru hex Sequence Bits 00000000 0000007f T0 7 00000080 000007FF T1 Tx 11 00000800 0000FFFF T2 Tx Tx 16 00010000 001FFFFF T3 Tx Tx Tx 21 00200000 03FFFFFF T4 Tx Tx Tx Tx 26 04000000 FFFFFFFF T5 Tx Tx Tx Tx Tx 32 [...] 4. All of the sequences synchronize on any byte that is not a Tx byte. If you are starting mid-run, skip initial Tx bytes. That will always be less than one character.
- ChrisSD 6y ago
  Note that UTF-8 has since been restricted to at most 4 bytes (i.e. the longest sequence is `T3 Tx Tx Tx`).
  - throwdbaaway 6y ago
    So now we know who is really responsible for the whole MySQL utf8mb4 fiasco -- these 2 guys sitting in a diner, conjuring up a brilliant scheme to cover 4 billions characters, which turned out to exceed the actual requirement by more than 2000x. September 1992: 2 guys scribbling on a placemat. January 1998: RFC 2279 defines UTF-8 to be between 1 to 6 bytes. March 2001: A bunch of CJK characters were added to Unicode Data 3.1.0, pushing the total to 94,140, exceeding the 16-bit limit of 3 bytes UTF-8. March 2002: MySQL added support for UTF-8, initially setting the limit to 6 bytes (https://github.com/mysql/mysql-server/commit/55e0a9c https://github.com/mysql/mysql-server/commit/55e0a9c) September 2002: MySQL decided to reduce the limit to 3 bytes, probably for storage efficiency reason (https://github.com/mysql/mysql-server/commit/43a506c https://github.com/mysql/mysql-server/commit/43a506c, https://adamhooper.medium.com/in-mysql-never-use-utf8-use-utf8mb4-11761243e434 https://adamhooper.medium.com/in-mysql-never-use-utf8-use-ut...) November 2003: RFC 3629 defines UTF-8 to be between 1 to 4 bytes. Arguably, if the placemat was smaller and the guys stopped at 4 bytes after running out of space, perhaps MySQL would have done the right thing? Ah, who am I kidding. The same commit would likely still happen. EDIT: Just notice this in the footnotes, and the plot thickens... > The 4, 5, and 6 byte sequences are only there for political reasons. I would prefer to delete these. So UTF-8 was indeed intended to be utf8mb3!
ChrisSD 6y ago
Other people have answered your question but I wanted to clarify one point. The word "character" here means "unicode code point". However, what the user thinks of as a single character can be made up of more than one code point. This presents a different problem and one UTF-8 itself can't help with. The Unicode Consortium has a report on extended grapheme clusters[0] (i.e. user-perceived characters). Essentially, if you're processing some text mid stream, it might not be clear if a code point is the start of a new user-perceived character or not. So you may want to skip ahead until an unambiguous symbol boundary is reached. [0]: https://www.unicode.org/reports/tr29/ https://www.unicode.org/reports/tr29/
fomine3 6y ago
It is called Self-synchronizing code, simple and beautiful design. https://en.wikipedia.org/wiki/Self-synchronizing_code https://en.wikipedia.org/wiki/Self-synchronizing_code

VMG 6y ago

https://web.archive.org/web/20210324065459/http://doc.cat-v.org/bell_labs/utf-8_history https://web.archive.org/web/20210324065459/http://doc.cat-v....

janvdberg 6y ago

More or less same the story but in an explainer video: 'Characters, Symbols and the Unicode Miracle - Computerphile' https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4

not_knuth 6y ago
This video is fantastic, especially if you've never really understood how Unicode worked before. Keeps it simple and gets to the point. I remember seeing this a couple of years ago and thinking "If only this existed when I learned about UTF-8, this would've saved me a lot of bad explanations and time". This is now my goto-video if anyone asks me how Unicode works.
- enriquto 6y ago
  > This video is fantastic, especially if you've never really understood how Unicode worked before. Can you people really learn things on videos? My brain sort of shuts down on audiovisual material, I can only really watch and understand light movies. For more complicated material, I can only learn it by reading. There's so much essential back and forth that is impossible on a video. With a text, you have everything already in front of you and your eyes and mind can wander freely. Maybe it's just me, but I really can't stand the fixed, inflexible rhythm that is imposed by listening to speech.
  - tux3 6y ago
    I also like text, but if you're genuinely asking, yes! Video evidently works great for a lot of people. You can't seek it as well as text, so instead rely on your memory more to remember things that are still unclear or don't make sense yet, and see when they are explained later. It's okay to have different learning styles I think, it's not too surprising to me that some prefer different medium
    - enriquto 6y ago
      It's precisely the seeking that kills me. The only interface for seeking a video is a tiny, one-dimensional bar. How do you remember exactly where the right information occurs? On the other hand, text is visible as a whole, and the "seeking" is two-dimensional. Much more efficient to seek. I have a good visual memory and I tend to remember exactly where on a page the formula that I want appears, and just glance away at it without moving any muscles other than my eyes. To seek a part on a video, the process is overwhelmingly more complicated. And then the place where I was before the seek is lost, and you have to seek for it again. Argh! Thanks for the answer, it turns out that we people are wildly different to each other.

TorKlingberg 6y ago

I think we should take a moment to appreciate how great UTF-8 is, and how it well it worked out. It's easy to get disillusioned with internet standards when IPv6 is taking forever and messaging is all proprietary locked down protocols. Yet character encodings used to be a horrible mess and now it's not. In the 90's the only practical solution was for everyone to use the same OS, same word processor, same web browser, and who cares about talking with foreigners anyway? I don't think it was always guaranteed to turn out well. China and Japan could have stayed with their own encodings. Microsoft and Apple could have done incompatible things. The tech world is full of bad things we're stuck with because there's no way to coordinate a change. Unicode has it's flaws, UTF-16 is still lurking here and there, everyone loves to argue about emoji, but overall text just works now.

roblabla 6y ago
> UTF-16 is still lurking here and there As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16/UCS2 as the string encoding. Also fun fact, on the Nintendo Switch, various subsystems use different kind of encoding. The filsystem submodule uses Shift-JIS, most of the other modules use UTF-8, but some others yet use UTF-16 (like the virtual keyboard, IIRC). A brilliant mess.
- ChrisSD 6y ago
  Personally I don't find UTF-16 to be too bad. It's a simple encoding and very easy to convert to/from UTF-8. So your program can be written in UTF-8 and your WinAPI wrappers can convert as/when needed.
  - jeltz 6y ago
    The bad thing with UTF-16 is that so much software assumes that one code point always is 16 bits.
    - cygx 6y ago
      Most of the hard stuff is there no matter the encoding (normalization, user-perceived characters spanning multiple code units, paths vs strings, ...).

bartwe 6y ago

Was always a bit surprised that if you had the length encoding byte, that it didn't also imply an offset by the values that should have been encoded with a shorter sequence

Tuna-Fish 6y ago
This is there to make decoding faster. I think it's a mistake.
- tacitusarc 6y ago
  The continuation bits also allow backwards traversal and proper null byte decoding.

0x0 6y ago

Interesting comment near the end, "2. The 4, 5, and 6 byte sequences are only there for political reasons. I would prefer to delete these." If that had happened, I guess emojis as we know them today might never have happened, since it would have limited us to 16 bits of code points. Or we would have had to start doing surrogate pairs even in UTF-8. Close call.

morelisp 6y ago
The 5-6 byte variants (and also 4 at the time) exist because of the need to round-trip UCS surrogate pairs through UTF-8, no? That's what I assume the "political reasons" are...
- deleted 6y ago
  [deleted]
- 0x0 6y ago
  I don't think so? Aren't UCS surrogate pairs at most 16bit each by their very purpose? Also, >16bit unicode code points came much later, I believe, in Unicode 2.0 in 1996 according to Wikipedia (vs UTF-8 which is from around 1992)
- ChrisSD 6y ago
  Surrogate pairs are only in UTF-16 so as to encode code points that require more than 16 bits. UTF-8 has no need of them because it's already a variable width encoding. If there were no code points larger than 16 bits then UTF-8 would only need a maximum of 3 bytes per code point and UTF-16 wouldn't need surrogate pairs. Well actually UTF-16 probably wouldn't exist at all because UCS-2 would have been enough for everybody.
  - deleted 6y ago
    [deleted]
- Sniffnoy 6y ago
  Others have already answered why surrogate pairs are irrelevant (and not UCS), but I think it's worth saying what the probable actual reason for 5-6 byte variants was. Remember that UCS and Unicode were at this point still two separate things; Unicode was supposed to be 16-bit (and later it got expanded, causing the whole surrogates mess), while UCS was supposed to be 31-bit. I assume the 5-6 byte variants were for UCS (back before it got merged with Unicode).

YesThatTom2 6y ago

ACM should put a plaque at that diner.

euroderf 6y ago
Yes but at what diner ? Wikipedia says NJ has about 525.
- YesThatTom2 6y ago
  We could ask. I've always assumed Prestige Diner & Restaurant just because it is close to the Murray Hill building.
  - v64 6y ago
    Another commenter stated it's Corner Café in New Providence [1] [1] https://news.ycombinator.com/item?id=26739049 https://news.ycombinator.com/item?id=26739049

dpcx 6y ago

Designing and coding up something of this importance in less than a week is impressive. Strangely, what I'm more impressed with is that, in 2003 - 11 years after all of this happened - Bell Labs still had _sendmail logs for two users of their system_. How much history do we throw away by trashing logs nowadays? Sure, there's probably a lot more traffic... but wow.

yiyus 6y ago
Plan 9 always had great archival utilities. I do not know what they used at those early days, but I guess that at the point rsc recovered it, it was stored in the venti server: https://p9f.org/sys/doc/venti/venti.html https://p9f.org/sys/doc/venti/venti.html

deleted 6y ago

[deleted]

haolez 6y ago

One of Rob's ideas - the GOPATH - is something that I enjoy very much and even port the concept to other languages, sometimes. However, it seems that the community as a whole didn't embrace it. I feel isolated, but in good company :D

BlueTemplar 6y ago
Please tell us more?
yobert 6y ago
I liked GOPATH at first, but the limitations of projects stepping on each other quickly made me sad about it. Now that go modules has removed the need for GOPATH, I'm very happy about it being obsolete. :)

peterdepasquale 6y ago

Those of us from NJ really need to know what diner though... > UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992.

kps 6y ago
“The diner was the Corner Café in New Providence, New Jersey.” — Rob Pike, https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-years-old-in-2012.html https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-y...

BlueTemplar 6y ago

The transition from 32 bit to 64 bit words was a missed opportunity to move from 8 bit to 32 bit bytes, which would have greatly simplified the fundamental aspect that is dealing with text in computers.

colejohnson66 6y ago
Some architectures use 8 bit bytes, but not really. What I mean is: some architectures require a word bigger than 8 bits (16+) be aligned. As in: to read a 32 bit word from memory, the first octet must have an address ending in two zero bits. But if you want to read only a byte, alignment doesn’t matter. IIRC, ARM used to have this “alignment” requirement.
- BlueTemplar 6y ago
  Why ? When ?

ChrisArchitect 6y ago

similar post of the same email/story, 15 days ago and a year ago here's more discussion from then: https://news.ycombinator.com/item?id=21212445 https://news.ycombinator.com/item?id=21212445

dang 6y ago

If curious, past threads: UTF-8 history (2003) - https://news.ycombinator.com/item?id=21212445 https://news.ycombinator.com/item?id=21212445 - Oct 2019 (52 comments) UTF-8 History (2003) - https://news.ycombinator.com/item?id=19565980 https://news.ycombinator.com/item?id=19565980 - April 2019 (3 comments) UTF-8 history - https://news.ycombinator.com/item?id=15236856 https://news.ycombinator.com/item?id=15236856 - Sept 2017 (1 comment) UTF-8 history (2003) - https://news.ycombinator.com/item?id=8648541 https://news.ycombinator.com/item?id=8648541 - Nov 2014 (7 comments) UTF-8 Original Proposal - https://news.ycombinator.com/item?id=6463466 https://news.ycombinator.com/item?id=6463466 - Sept 2013 (3 comments) UTF-8 History - https://news.ycombinator.com/item?id=2081932 https://news.ycombinator.com/item?id=2081932 - Jan 2011 (2 comments) The history of UTF-8 as told by Rob Pike - https://news.ycombinator.com/item?id=577116 https://news.ycombinator.com/item?id=577116 - April 2009 (1 comment) --- Don't miss this great link from the 2017 thread: https://www.flickr.com/photos/ajstarks/sets/72157631470798870 https://www.flickr.com/photos/ajstarks/sets/7215763147079887...

pierrebai 6y ago

One thing to realize is that UTF-8 is here and required only because we're all stuck with limitations based on the low-power, low-limitations of early systems. There is not a single reason, other than historical, backward compatibility and the overall ecosystem putting incommensurate pressure on all future systems, why the basic memory unit is 8 bits. We're stuck on 8 bits because all API, OS and specifications assume the basic units is 8 bits. If computer memory had evolved faster, or computer usage spread slower, the basic unit could have been 32 bits, or even 64 bits. Think how much easier things would be. UTF could be encoded as a single unit, without ambiguous order. What we've lost due to byte and UTF-8 is for example the text editor as the universal editor. Due to the need of UTF-8 decoding, we're no longer able to open a any (possibly binary) file as text and save it reliably after editing one character. There are other consequences, for example at the file-system level for file names, corruption, how to handle invalid UTF-8 sequences, etc. (And before you talk about the byte order problem, that is the point: there is no order problem for the basic unit you choose. No one (outside of some low-level 1-bit serial interfaces) cars about bit order or nibble order of bytes. Because the bit order is hidden within all interface accepting byte as the unit.)

cesarb 6y ago
It's hard to argue against a counterfactual, but I believe there is a "sweet spot" for the size of the basic memory unit, and that "sweet spot" is probably in the range from 4 to 16 bits (power of two sizes are desirable, so the possible sizes would be 4 bits, 8 bits, and 16 bits), so even if computers evolved from scratch today already starting with large memories, the basic unit of memory addressing still wouldn't be that big. My guess is that the chosen size would be 16 bits.

cryptonector 6y ago

> helix: Sep 8 03:22:13: ken: upas/sendmail: remote inet!xopen.co.uk!xojig > From ken Tue Sep 8 03:22:07 EDT 1992 (xojig@xopen.co.uk) 6833 It was 1992 and they were still using UUCP-style addressing.