14 ms·

UTF-8 is a brilliant design

• ceh56 1y ago
  Another collaboration by Pike and Thompson can be seen here: https://go.dev/ https://go.dev/.
• happytoexplain 1y ago
  I have a love-hate relationship with backwards compatibility. I hate the mess - I love when an entity in a position of power is willing to break things in the name of advancement. But I also love the cleverness - UTF-8, UTF-16, EAN, etc. To be fair, UTF-8 sacrifices almost nothing to achieve backwards compat though.
  • mort96 1y ago
    Yeah I honestly don't know what I would change. Maybe replace some of the control characters with more common characters to save a tiny bit of space, if we were to go completely wild and break Unicode backward compatibility too. As a generic multi byte character encoding format, it seems completely optimal even in isolation.
  • amluto 1y ago
    > To be fair, UTF-8 sacrifices almost nothing to achieve backwards compat though. It sacrifices the ability to encode more than 21 bits, which I believe was done for compatibility with UTF-16: UTF-16’s awful “surrogate” mechanism can only express code units up to 2^21-1. I hope we don’t regret this limitation some day. I’m not aware of any other material reason to disallow larger UTF-8 code units.
    • 1oooqooq 1y ago
      the limitation tomorrow will be today's implementations, sadly.
    • throw0101d 1y ago
      > It sacrifices the ability to encode more than 21 bits, which I believe was done for compatibility with UTF-16: UTF-16’s awful “surrogate” mechanism can only express code units up to 2^21-1z Yes, it is 'truncated' to the "UTF-16 accessible range": * https://datatracker.ietf.org/doc/html/rfc3629#section-3 https://datatracker.ietf.org/doc/html/rfc3629#section-3 * https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History Thompson's original design could handle up to six octets for each letter/symbol, with 31 bits of space: * https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
      • gpvos 1y ago
        You could even extend UTF-8 to make 0xFE and 0xFF valid starting bytes, with 6 and 7 following bytes each, and get 42 bits of space. I seem to remember Perl allowed that for a while in its v-strings notation. Edit: just tested this, Perl still allows this, but with an extra twist: v-notation goes up to 2^63-1. From 2^31 to 2^36-1 is encoded as FE + 6 bytes, and everything above that is encoded as FF + 12 bytes; the largest value it allows is v9223372036854775807, which is encoded as FF 80 87 BF BF BF BF BF BF BF BF BF BF. It probably doesn't allow that one extra bit because v-notation doesn't work with negative integers.
    • mort96 1y ago
      That isn't really a case of UTF-8 sacrificing anything to be compatible with UTF-16. It's Unicode, not UTF-8 that made the sacrifice: Unicode is limited to 21 bits due to UTF-16. The UTF-8 design trivially extends to support 6 byte long sequences supporting up to 31-bit numbers. But why would UTF-8, a Unicode character encoding, support code points which Unicode has promised will never and can never exist?
      • MyOutfitIsVague 1y ago
        In an ideal future (read: fantasy), utf-16 gets formally deprecated and trashed, freeing the surrogate sequences and full range for utf-8. Or utf-16 is officially considered a second class citizen, and some code points are simply out of its reach.
      • GuB-42 1y ago
        Is 21 bits really a sacrifice. It is 2 million codepoints, we currently use about a tenth of that. Even with all Chinese characters, de-unified, all the notable historical and constructed scripts, technical symbols, and all the submitted emoji, including rejections, you are still way short of a million. We are probably never need more than 21 bits unless we start stretching the definition of what text is.
        • moefh 1y ago
          It's not 2 million, it's a little over 1 million. The exact number is 1112064 = 2^16 - 2048 + 16*2^16: in UTF-16, 2 bytes can encode 2^16 - 2048 code points, and 4 bytes can encode 16*2^16 (the 2048 surrogates are not counted because they can never appear by themselves, they're used purely for UTF-16 encoding).
          • chuckadams 1y ago
            Even with just 1 million codepoints, why did they feel the need for CJK unification? Was it so it would all fit in UCS-2 or something?
    • Analemma_ 1y ago
      It's always dangerous to stick one's neck out and say "[this many bits] ought to be enough for anybody", but I think it's very unlikely we'll ever run out of UTF-8 sequences. UTF-8 can represent about 1.1 million code points, of which we've assigned about 160,000 actual characters, plus another ~140,000 in the Private Use Area, which won't expand. And that's after getting nearly all of the world's known writing systems: the last several Unicode updates have added a few thousand characters here and there for very obscure and/or ancient writing systems, but those won't go on forever (and things like emojis rarely only get a handful of new code points per update, because most new emojis are existing code points with combining characters). If I had to guess, I'd say we'll run out of IPv6 addresses before we run out of unassigned UTF-8 sequences.
      • lyu07282 1y ago
        The oldest script in unicode, sumerian cuneiform, is ~5,200 years old, if we were to invent new scripts at the same rate we would hit 1.1 million code points in around 31,000 years. So yeah nothing to worry about, you are absolutely right. Unless we join some intergalactic federation of planets, although they probably already have their own encoding standards we could just adopt.
    • layer8 1y ago
      That limitation will be trivial to lift once UTF-16 compatibility can be disregarded. This won’t happen soon, of course, given JavaScript and Windows, but the situation might be different in a hundred or thousand years. Until then, we still have a lot of unassigned code points. In addition, it would be possible to nest another surrogate-character-like scheme into UTF-16 to support a larger character set.
    • cryptonector 1y ago
      > It sacrifices the ability to encode more than 21 bits No, UTF-8's design can encode up to 31 bits of codepoints. The limitation to 21 bits comes from UTF-16, which was then adopted for UTF-8 too. When UTF-16 dies we'll be able to extend UTF-8 (well, compatibility will be a problem).
  • procaryote 1y ago
    > I love when an entity in a position of power is willing to break things in the name of advancement. It's less fun when you have things that need to keep working break because someone felt like renaming a parameter, or that a part of the standard library looks "untidy"
    • happytoexplain 1y ago
      I agree! And yet I lovingly sacrifice my man-hours to it when I decide to bump that major version number in my dependency manifest.
      • account42 1y ago
        The key words here being "I decide". I'm going to express a lot less love when someone else decides.
      • procaryote 1y ago
        Or minor versions of python... Honestly python is probably one of the worst offender in this as they combine happily making breaking changes for low value rearranging of deck chairs with a dynamic language where you might only find out in runtime. The fact that they've also decided to use an unconventional intepretation of minor version shows how little they care.
        • chuckadams 1y ago
          The term "semantic versioning" didn't even exist until 2010, which is well after the birth of Python. Sure, it semi-formalized a convention from long before, but it was hardly universal.
          • account42 1y ago
            The ideals behind semantic versioning existed long before the marketing term.
          • procaryote 1y ago
  • cryptonector 1y ago
    > To be fair, UTF-8 sacrifices almost nothing to achieve backwards compat though. There were apps that completely rejected non-7-bit data back in the day. Backwards compatibility wasn't the only point. The point of UTF-8 is more (IMO) that UTF-32 is too bulky, UCS-2 was insufficient, UTF-16 was an abortion, and only UTF-8 could have the right trade-offs.
• 3pt14159 1y ago
  I remember a time before UTF-8's ubiquity. It was such a headache moving to i18z. I love UTF-8.
  • glxxyz 1y ago
    I worked on an email client. Many many character set headaches.
  • linguae 1y ago
    I remember learning Japanese in the early 2000s and the fun of dealing with multiple encodings for the same language: JIS, Shift-JIS, and EUC. As late as 2011 I had to deal with processing a dataset encoded under EUC in Python 2 for a graduate-level machine learning course where I worked on a project for segmenting Japanese sentences (typically there are no spaces in Japanese sentences). UTF-8 made processing Japanese text much easier! No more needing to manually change encoding options in my browser! No more mojibake!
    • deleted 1y ago
      [deleted]
    • layer8 1y ago
      On the other hand, you now have to deal with the issues of Han unification: https://en.wikipedia.org/wiki/Han_unification#Examples_of_language-dependent_glyphs https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
    • pezezin 1y ago
      I live in Japan and I still receive the random email or work document encoded in Shit-JIS. Mojibake is not as common as it once was, but still a problem.
      • rmunn 1y ago
        I'm assuming you misspelled Shift-JIS on purpose because you're sick and tired of dealing with it. If that was an accidental misspelling, it was inspired. :-)
• bruce511 1y ago
  While the backward compatibility of utf-8 is nice, and makes adoption much easier, the backward compatibility does not come at any cost to the elegance of the encoding. In other words, yes it's backward compatible, but utf-is also compact and elegant even without that.
  • nextaccountic 1y ago
    UTF-8 also enables this mindblowing design for small string optimization - if the string has 24 bytes or less it is stored inline, otherwise it is stored on the heap (with a pointer, a length, and a capacity - also 24 bytes) https://github.com/ParkMyCar/compact_str https://github.com/ParkMyCar/compact_str How cool is that (Discussed here https://news.ycombinator.com/item?id=41339224 https://news.ycombinator.com/item?id=41339224)
    • adgjlsfhk1 1y ago
      How is that UTF8 specific?
      • ubitaco 1y ago
        It's slightly buried in the readme on Github: > how can we store a 24 byte long string, inline? Don't we also need to store the length somewhere? > To do this, we utilize the fact that the last byte of our string could only ever have a value in the range [0, 192). We know this because all strings in Rust are valid UTF-8, and the only valid byte pattern for the last byte of a UTF-8 character (and thus the possible last byte of a string) is 0b0XXXXXXX aka [0, 128) or 0b10XXXXXX aka [128, 192)
        • Dylan16807 1y ago
          Any Unicode encoding would allow that. UTF-32 has an entire spare byte to put flags into. 24 or 21 bit encodings have spare bits that could act as flags. UTF-16 has plenty of invalid code units, or you could use a high surrogate in the last 2 bytes as your flag.
      • deleted 1y ago
• quectophoton 1y ago
  Having the continuation bytes always start with the bits `10` also make it possible to seek to any random byte, and trivially know if you're at the beginning of a character or at a continuation byte like you mentioned, so you can easily find the beginning of the next or previous character. If the characters were instead encoded like EBML's variable size integers[1] (but inverting 1 and 0 to keep ASCII compatibility for the single-byte case), and you do a random seek, it wouldn't be as easy (or maybe not even possible) to know if you landed on the beginning of a character or in one of the `xxxx xxxx` bytes. [1]: https://www.rfc-editor.org/rfc/rfc8794#section-4.4 https://www.rfc-editor.org/rfc/rfc8794#section-4.4
  • 1oooqooq 1y ago
    so you replace one costly sweeping with a costly sweeping. i wouldn't call that an advantage in any way over junping n bytes. what you describe is the bare minimum so you even know what you are searching for while you scan pretty much everything every time.
    • hk__2 1y ago
      What do you mean? What would you suggest instead? Fixed-length encoding? It would take a looot of space given all the character variations you can have.
      • gertop 1y ago
        UTF-16 is both simpler to parse and more compact than utf-8 when writing non-english characters. UTF-8 didn't win on technical merits, it won becausw it was mostly backwards compatible with all American software that previously used ASCII only. When you leave the anglosphere you'll find that some languages still default to other encodings due to how large utf-8 ends up for them (Chinese and Japanese, to name two).
        • ISV_Damocles 1y ago
          UTF-16 is also just as complicated as UTF-8 requiring multibyte characters to cover the entirety of Unicode, so it doesn't avoid the issue you're complaining about for the newest languages added, and it has the added complexity of a BOM being required to be sure you have the pairs of bytes in the right order, so you are more vulnerable to truncated data being unrecoverable versus UTF-8. UTF-32 would be a fair comparison, but it is 4 bytes per character and I don't know what, if anything, uses it.
• twbarr 1y ago
  It should be noted that the final design for UTF-8 was sketched out on a placemat by Rob Pike and Ken Thompson.
  • hu3 1y ago
    I wonder if that placemat still exists today. It would be such an important piece of computer history.
    • ot 1y ago
      > It was so easy once we saw it that there was no reason to keep the placemat for notes, and we left it behind. Or maybe we did bring it back to the lab; I'm not sure. But it's gone now. https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-years-old-in-2012.html https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-y...
• cyberax 1y ago
  UTF-8 is simply genius. It entirely obviated the need for clunky 2-byte encodings (and all the associated nonsense about byte order marks). The only problem with UTF-8 is that Windows and Java were developed without knowledge about UTF-8 and ended up with 16-bit characters. Oh yes, and Python 3 should have known better when it went through the string-bytes split.
  • wrs 1y ago
    UTF-16 made lots of sense at the time because Unicode thought "65,536 characters will be enough for anybody" and it retains the 1:1 relationship between string elements and characters that everyone had assumed for decades. I.e., you can treat a string as an array of characters and just index into it with an O(1) operation. As Unicode (quickly) evolved, it turned out not that only are there WAY more than 65,000 characters, there's not even a 1:1 relationship between code points and characters, or even a single defined transformation between glyphs and code points, or even a simple relationship between glyphs and what's on the screen. So even UTF-32 isn't enough to let you act like it's 1980 and str[3] is the 4th "character" of a string. So now we have very complex string APIs that reflect the actual complexity of how human language works...though lots of people (mostly English-speaking) still act like str[3] is the 4th "character" of a string. UTF-8 was designed with the knowledge that there's no point in pretending that string indexing will work. Windows, MacOS, Java, JavaScript, etc. just missed the boat by a few years and went the wrong way.
    • rowls66 1y ago
      I think more effort should have been made to live with 65,536 characters. My understanding is that codepoints beyond 65,536 are only used for languages that are no longer in use, and emojis. I think that adding emojis to unicode is going to be seen a big mistake. We already have enough network bandwith to just send raster graphics for images in most cases. Cluttering the unicode codespace with emojis is pointless.
      • dudeinjapan 1y ago
        CJK unification (https://en.wikipedia.org/wiki/CJK_Unified_Ideographs https://en.wikipedia.org/wiki/CJK_Unified_Ideographs) i.e. combining "almost same" Chinese/Japanese/Korean characters into the same codepoint, was done for this reason, and we are now living with the consequence that we need to load separate Traditional/Simplified Chinese, Japanese, and Korean fonts to render each language. Total PITA for apps that are multi-lingual.
• hyperman1 1y ago
  One thing I always wonder: It is possible to encode a unicode codepoint with too much bytes. UTF-8 forbids these, only the shortest one is valid. E.g 00000001 is the same as 11000000 10000001. So why not make the alternatives impossible by adding the start of the last valid option? So 11000000 10000001 would give codepoint 128+1 as values 0 to 127 are already covered by a 1 byte sequence. The advantages are clear: No illegal codes, and a slightly shorter string for edge cases. I presume the designers thought about this, so what were the disadvantages? The required addition being an unacceptable hardware cost at the time? UPDATE: Last bitsequence should of course be 10000001 and not 00000001. Sorry for that. Fixed it.
  • deleted 1y ago
    [deleted]
  • rhet0rica 1y ago
    See quectophoton's comment—the requirement that continuation bytes are always tagged with a leading 10 is useful if a parser is jumping in at a random offset—or, more commonly, if the text stream gets fragmented. This was actually a major concern when UTF-8 was devised in the early 90s, as transmission was much less reliable than it is today.
    • rhet0rica 1y ago
      Addendum: This was posted to the front page today: https://doc.cat-v.org/bell_labs/utf-8_history https://doc.cat-v.org/bell_labs/utf-8_history It also notes that UTF-8 protects against the dangers of NUL and '/' appearing in filenames, which would kill C strings and DOS path handling, respectively.
  • School-Cotton 1y ago
    Because then it would be impossible to tell from looking at a byte whether it is the beginning of a character or not, which is a useful property of UTF-8.
  • variadix 1y ago
    https://en.m.wikipedia.org/wiki/Self-synchronizing_code https://en.m.wikipedia.org/wiki/Self-synchronizing_code
• rmccue 1y ago
  Love the UTF-8 playground that's linked: https://utf8-playground.netlify.app/ https://utf8-playground.netlify.app/ Would be great if it was possible to enter codepoints directly; you can do it via the URL (`/F8FF` eg), but not in the UI. (Edit, the future is now. https://github.com/vishnuharidas/utf8-playground/pull/6 https://github.com/vishnuharidas/utf8-playground/pull/6)
  • vishnuharidas 1y ago
    Thanks for the contribution, this is now merged and live.
• alberth 1y ago
  I’ve re-read so many times Joel’s article on Unicode. It’s also very helpful. https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
  • mixmastamyk 1y ago
    Read that a few times back then as well, but that and other pieces of the day never told you how to actually write a program that supported Unicode. Just facts about it. So I went around fixing UnicodeErrors in Python at random, for years, despite knowing all that stuff. It wasn't until I read Batchelder's piece on the "Unicode Sandwich," about a decade later that I finally learned how to write a program to support it properly, rather than playing whack-a-mole.
    • rmunn 1y ago
      > ... Batchelder's piece on the "Unicode Sandwich," ... Is this the piece you mean? https://nedbatchelder.com/text/unipain.html https://nedbatchelder.com/text/unipain.html
      • mixmastamyk 1y ago
        I think so, looks like it was from a presentation.
    • mixmastamyk 1y ago
      ^Necessary but not sufficient.
• burtekd 1y ago
  I'm just gonna leave this here too: https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4
• quotemstr 1y ago
  Great example of a technology you get from a brilliant guy with a vision and that you'll never get out of a committee.
• wetpaws 1y ago
  [dead]
• librasteve 1y ago
  some insightful unicode regex examples... https://dev.to/bbkr/utf-8-internal-design-5c8b https://dev.to/bbkr/utf-8-internal-design-5c8b
  • hamburglar 1y ago
    Regex? Did you link to the wrong page? I see no regexes on that page.
    • librasteve 1y ago
      well you have to click around a bit and be prepared to look at the other pages in Pabels series of posts … I linked to this one since I felt it chimes well with the OP
• nostrademons 1y ago
  For more on UTF-8's design, see Russ Cox's one-pager on it: https://research.swtch.com/utf8 https://research.swtch.com/utf8 And Rob Pike's description of the history of how it was designed: https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
  • jjice 1y ago
    Thank you for posting these - the Bell Labs crew is just a different breed. Of course it's Pike and Thompson and the gang. The amount of contributions these guys made to the world of computing is insane.
  • extraduder_ire 1y ago
    I consider "designed on a placemat" to be a selling point for any high-quality standard that will last.
• twoodfin 1y ago
  UTF-8 is indeed a genius design. But of course it’s crucially dependent on the decision for ASCII to use only 7 bits, which even in 1963 was kind of an odd choice. Was this just historical luck? Is there a world where the designers of ASCII grabbed one more bit of code space for some nice-to-haves, or did they have code pages or other extensibility in mind from the start? I bet someone around here knows.
  • colejohnson66 1y ago
    The idea was that the free bit would be repurposed, likely for parity.
    • EGreg 1y ago
      I would love to think this is true, and it makes sense, but do you have any actual evidence for this you could share with HN?
    • KPGv2 1y ago
      This is not true. ASCII (technically US-ASCII) was a fixed-width encoding of 7 bits. There was no 8th bit reserved. You can read the original standard yourself here: https://ia600401.us.archive.org/23/items/enf-ascii-1968-1970/Image070917151315.pdf https://ia600401.us.archive.org/23/items/enf-ascii-1968-1970... Crucially, "the 7-bit coded character set" is described on page 6 using only seven total bits (1-indexed, so don't get confused when you see b7 in the chart!). There is an encoding mechanism to use 8 bits, but it's for storage on a type of magnetic tape, and even that still is silent on the 8th bit being repurposed. It's likely, given the lack of discussion about it, that it was for ergonomic or technical purposes related to the medium (8 is a power of 2) rather than for future extensibility.
      • kbolino 1y ago
        Notably, it is mentioned that the 7-bit code is developed "in anticipation of" ISO requesting such a code, and we see in the addenda attached at the end of the document that ISO began to develop 8-bit codes extending the base 7-bit code shortly after it was published. So, it seems that ASCII was kept to 7 bits primarily so "extended ASCII" sets could exist, with additional characters for various purposes (such as other languages, but also for things like mathematical symbols).
• sheerun 1y ago
  I'll mention IPv6 as bad design that could have been potentially UTF-8-like success story
  • tialaramex 1y ago
    No. UTF-8 is for encoding text, so we don't need to care about it being variable length because text was already variable length. The network addresses aren't variable length, so if you decide "Oh IPv6 is variable length" then you're just making it worse with no meaningful benefit. The IPv4 address is 32 bits, the IPv6 address is 128 bits. You could go 64 but it's much less clear how to efficiently partition this and not regret whatever choices you do make in the foreseeable future. The extra space meant IPv6 didn't ever have those regrets. It suits a certain kind of person to always pay $10M to avoid the one-time $50M upgrade cost. They can do this over a dozen jobs in twenty years, spending $200M to avoid $50M cost and be proud of saving money.
    • sheerun 1y ago
      You reserve 32 bits of these 128 just like UTF-8 did for theirs for ASCII for backward-compatibility, and request backward compatible fall-back from user interfaces, I hope it clears it
• zamalek 1y ago
  Even for varints (you could probably drop the intermediate prefixes for that). There are many examples of using SIMD to decode utf-8, where-as the more common protobuf scheme is known to be hostile to SIMD and the branch predictor.
  • camel-cdr 1y ago
    Yeah, protobuf's varint are quite hard to decode with current SIMD instructions, but it would be quite easy, if we get element wise pext/pdep instructions in the future. (SVE2 already has those, but who has SVE2?)
• postalrat 1y ago
  Looks similar to midi
• carlos256 1y ago
  No, it's not. It's just a form of Elias-Gamma coding.
  • carlos256 1y ago
    * unary encoding coding.
• dotslashmain 1y ago
  Rob Pike and Ken Thompson are brilliant computer scientists & engineers.
  • carlos256 1y ago
    [flagged]
• dpc_01234 1y ago
  UTF-8 is a undeniably a good answer, but to a relatively simple bit twiddling / variable len integer encoding problem in a somewhat specific context. I realize that hindsight is 20/20, and time were different, but lets face it: "how to use an unused top bit to best encode larger number representing Unicode" is not that much of challenge, and the space of practical solutions isn't even all that large.
  • ivanjermakov 1y ago
    I just realised that all latin text is wasting 12% of storage/memory/bandwidth with MSB zero. At least is compresses well. Are there any technology that utilizes 8th bit for something useful, e.g. error checking?
    • tmiku 1y ago
      See mort96's comments about 7-bit ASCII and parity bits (https://news.ycombinator.com/item?id=45225911 https://news.ycombinator.com/item?id=45225911). Kind of archaic now, though - 8-bit bytes with the error checking living elsewhere in the stack seems to be preferred.
  • Tuna-Fish 1y ago
    Except that there were many different solutions before UTF-8, all of which sucked really badly. UTF-8 is the best kind of brilliant. After you've seen it, you (and I) think of it as obvious, and clearly the solution any reasonable engineer would come up with. Except that it took a long time for it to be created.
• ummonk 1y ago
  > Another one is the ISO/IEC 8859 encodings are single-byte encodings that extend ASCII to include additional characters, but they are limited to 256 characters. ISO 2022 allowed you to use control codes to switch between ISO 8859 character sets though, allowing for mixed script text streams.
• Dwedit 1y ago
  Meanwhile Shift-JIS has a bad design, since the second byte of a character can be any ASCII character 0x40-0x9E. This includes brackets, backslash, caret, backquote, curly braces, pipe, and tilde. This can cause a path separator or math operator to appear in text that is encoded as Shift-JIS but interpreted as plain ASCII. UTF-8 basically learned from the mistakes of previous encodings which allowed that kind of thing.
• tiahura 1y ago
  How many llm tokens are wasted everyday resolving utf issues?
• vintermann 1y ago
  UTF-8 is as good as a design as could be expected, but Unicode has scope creep issues. What should be in Unicode? Coming at it naively, people might think the scope is something like "all sufficiently widespread distinct, discrete glyphs used by humans for communication that can be printed". But that's not true, because * It's not discrete. Some code points are for combining with other code points. * It's not distinct. Some glyphs can be written in multiple ways. Some glyphs which (almost?) always display the same, have different code points and meanings. * It's not all printable. Control characters are in there - they pretty much had to be due to compatibility with ASCII, but they've added plenty of their own. I'm not aware of any Unicode code points that are animated - at least what's printable, is printable on paper and not just on screen, there are no marquee or blink control characters, thank God. But, who knows when that invariant will fall too. By the way, I know of one utf encoding the author didn't mention, utf-7. Like utf-8, but assuming that the last bit wasn't safe to use (apparently a sensible precaution over networks in the 80s). My boss managed to send me a mail encoded in utf-7 once, that's how I know what it is. I don't know how he managed to send it, though.
  • syncsynchalt 1y ago
    UTF-7 is/was mostly for email, which is not an 8-bit clean transport. It is obsolete and can't encode supplemental planes (except via surrogate pairs, which were meant for UTF-16). There is also UTF-9, from an April Fools RFC, meant for use on hosts with 36-bit words such as the PDP-10.
    • syncsynchalt 1y ago
      I meant to specify, the aim of UTF-7 is better performed by using UTF-8 with `Content-Transfer-Encoding: quoted-printable`
  • Cloudef 1y ago
    Indeed, one pain point of unicode is CJK unification. https://heistak.github.io/your-code-displays-japanese-wrong/ https://heistak.github.io/your-code-displays-japanese-wrong/
• anthonyiscoding 1y ago
  UTF-8 contributors are some of our modern day unsung heroes. The design is brilliant but the dedication to encode every single way humans communicate via text into a single standard, and succeed at it, is truly on another level. Most other standards just do the xkcd thing: "now there's 15 competing standards"
• sawyna 1y ago
  I have always wondered - what if the utf-8 space is filled up? Does it automatically promote to having a 5th byte? Is that part of the spec? Or are we then talking about utf-16?
  • KingLancelot 1y ago
    [dead]
  • kzrdude 1y ago
    utf-8 is just an encoding of unicode. UTF-8 is specified in a way so that it can encode all unicode codepoints up to 0x10FFFF. It doesn't extend further. And UTF-16 also encodes unicode in a similar same way, it doesn't encode anything more. So what would need to happen first would be that unicode decides they are going to include larger codepoints. Then UTF-8 would need to be extended to handle encoding them. (But I don't think that will happen.) It seems like Unicode codepoints are less than 30% allocated, roughly. So there's 70% free space.. --- Think of these three separate concepts to make it clear. We are effectively dealing with two translations - one from the abstract symbol to defined unicode code point. Then from that code point we use UTF-8 to encode it into bytes. 1. The glyph or symbol ("A") 2. The unicode code point for the symbol (U+0041 Latin Capital Letter A) 3. The utf-8 encoding of the code point, as bytes (0x41)
    • duskwuff 1y ago
      As an aside: UTF-8, as originally specified in RFC 2279, could encode codepoints up to U+7FFFFFFF (using sequences of up to six bytes). It was later restricted to U+10FFFF to ensure compatibility with UTF-16.
  • vishnuharidas 1y ago
    UTF-8 can represent up to 1,114,112 characters in Unicode. And in Unicode 15.1 (2023, https://www.unicode.org/versions/Unicode15.1.0/ https://www.unicode.org/versions/Unicode15.1.0/) a total of 149,813 characters are included, which covers most of the world's languages, scripts, and emojis. That leaves a 960K space for future expansion. So, it won't fill up during our lifetime I guess.
• TacticalCoder 1y ago
  [dead]
• billforsternz 1y ago
  A little off topic but amidst a lot of discussion of UTF-8 and its ASCII compatibility property I'm going to mention my one gripe with ASCII, something I never see anyone talking about, something I've never talked about before: The damn 0x7f character. Such an annoying anomaly in every conceivable way. It would be much better if it was some other proper printable punctuation or punctuation adjacent character. A copyright character. Or a pi character or just about anything other than what it already is. I have been programming and studying packet dumps long enough that I can basically convert hex to ASCII and vice versa in my head but I still recoil at this anomalous character (DELETE? is that what I should call it?) every time.
  • kragen 1y ago
    Much better in every way except the one that mattered most: being able to correct punching errors in a paper tape without starting over. I don't know if you have ever had to use White-Out to correct typing errors on a typewriter that lacked the ability natively, but before White-Out, the only option was to start typing the letter again, from the beginning. 0x7f was White-Out for punched paper tape: it allowed you to strike out an incorrectly punched character so that the message, when it was sent, would print correctly. ASCII inherited it from the Baudot–Murray code. It's been obsolete since people started punching their tapes on computers instead of Teletypes and Flexowriters, so around 01975, and maybe before; I don't know if there was a paper-tape equivalent of a duplicating keypunch, but that would seem to eliminate the need for the delete character. Certainly TECO and cheap microcomputers did.
    • billforsternz 1y ago
      Nice, thanks.
  • Agraillo 1y ago
    Related: Why is there a “small house” in IBM's Code page 437? (glyphdrawing.club) [1]. There are other interesting articles mentioned in the discussion. m_walden probably would comment here himself [1] https://news.ycombinator.com/item?id=43667010 https://news.ycombinator.com/item?id=43667010
• fmajid 1y ago
  Well, yes, Ken Thompson, the father of Unix, is behind it.
• blindriver 1y ago
  It took time for UTF-8 to make sense. Struggling with how large everything was was a real problem just after the turn of the century. Today it makes more sense because capacity and compute power is much greater but back then it was a huge pain in the ass.
  • gpvos 1y ago
    It made much more sense than UTF-16 or any of the existing multi-byte character sets, and the need for more than 256 characters had been apparent for decades. Seeing its simplicity, it made perfect sense almost immediately.
    • blindriver 1y ago
      No, it didn't. Not at the time. Like I said processing and storage were a pain back around the 2000-ish time. Windows supported UCS-2 (predecessor to UTF-16) which was fixed width 16-bit and faster to decode and encode, and since most of the world was Windows at the time, it made more sense to use UCS-2. Also, the world was only beginning to be more connected so UTF-8 seemed overkill. NOW in hindsight it makes more sense to use UTF-8 but it wasn't clear back 20 years ago it was worth it.
      • gpvos 1y ago
        Maybe if you were entrenched in the Windows world. Storage-wise, UTF-8 is usually better since so much data is ASCII with maybe the occasional accented character. The speed issue only really matters to Windows NT since that was UCS-2 inside, but it wasn't a problem for many.
      • acdha 1y ago
        The need was clear even 30 years ago when UTF-16 was standardized in 1996. UCS-2 was known at the time to be inadequate but there was a period from the mid-80s to early 90s where western developers tried to rpetend that they could only support a tiny fraction of Asian languages like Chinese (>50k characters, even if Han unification was uncontroversial), scholarly and technical usage, etc. The language used in 1988 was “Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988)” with the idea that other characters could be punted into a private registry. Once enough people accepted that this approach was impractical, UCS-2 was replaced with UTF-16 and surrogate codes. At that point it was clear that UTF-8 was better in almost every scenario because neither had an advantage for random access and UTF-8 was usually substantially smaller. 1. https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf
• LorenPechtel 1y ago
  Now fix fonts! It should be possible to render any valid string in a font.
• fleebee 1y ago
  If you want to delve deeper into this topic and like the Advent of Code format, you're in luck: i18n-puzzles[1] has a bunch of puzzles related to text encoding that drill how UTF-8 (and other variants such as UTF-16) work into your brain. [1]: https://i18n-puzzles.com/ https://i18n-puzzles.com/
• mikelabatt 1y ago
  Nice article, thank you. I love UTF-8, but I only advocate it when used with a BOM. Otherwise, an application may have no way of knowing that it is UTF-8, and that it needs to be saved as UTF-8. Imagine selecting New/Text Document in an environment like File Explorer on Windows: if the initial (empty) file has a BOM, any app will know that it is supposed to be saved again as UTF-8 once you start working on it. But with no BOM, there is no such luck, and corruption may be just around the corner, even when the editor tries to auto-detect the encoding (auto-detection is never easy or 100% reliable, even for basic Latin text with "special" characters) The same can happen to a plain ASCII file (without a BOM): once you edit it, and you add, say, some accented vowel, the chaos begins. You thought it was Italian, but your favorite text editor might conclude it's Vietnamese! I've even seen Notepad switch to a different default encoding after some Windows updates. So, UTF-8 yes, but with a BOM. It should be the default in any app and operating system.
  • Cloudef 1y ago
    BOM is awful as it breaks concatenation. In modern world everything should be just assumed to be UTF8 by default.
  • rmunn 1y ago
    The fact that you advocate using a BOM with UTF-8 tells me that you run Windows. Any long-term Unix user has probably seen this error message before (copy and pasted from an issue report I filed just 3 days ago): bash: line 1: #!/bin/bash: No such file or directory If you've got any experience with Linux, you probably suspect the problem already. If your only experience is with Windows, you might not realize the issue. There's an invisible U+FEFF lurking before the `#!`. So instead of that shell script starting with the `#!` character pair that tells the Linux kernel "The application after the `#!` is the application that should parse and run this file", it actually starts with `<FEFF>#!`, which has no meaning to the kernel. The way this script was invoked meant that Bash did end up running the script, with only one error message (because the line did not start with `#` and therefore it was not interpreted as a Bash comment) that didn't matter to the actual script logic. This is one of the more common problems caused by putting a BOM in UTF-8 files, but there are others. The issue is that adding a BOM, as can be seen here, *breaks the promise of UTF-8*: that a UTF-8 file that contains only codepoints below U+007F can be processed as-is, and legacy logic that assumes ASCII will parse it correctly. The Linux kernel is perfectly aware of UTF-8, of course, as is Bash. But the kernel logic that looks for `#!`, and the Bash logic that look for a leading `#` as a comment indicator to ignore the line, do *not* assume a leading U+FEFF can be ignored, nor should they (for many reasons). What should happen is that these days, every application should assume UTF-8 if it isn't informed of the format of the file, unless and until something happens to make it believe it's a different format (such as reading a UTF-16 BOM in the first two bytes of the file). If a file fails to parse as UTF-8 but there are clues that make another encoding sensible, reparsing it as something else (like Windows-1252) might be sensible. But putting a BOM in UTF-8 causes more problems than it solves, because it *breaks* the fundamental promise of UTF-8: ASCII compatibility with Unicode-unaware logic.
• modeless 1y ago
  UTF-8 is great and I wish everything used it (looking at you JavaScript). But it does have a wart in that there are byte sequences which are invalid UTF-8 and how to interpret them is undefined. I think a perfect design would define exactly how to interpret every possible byte sequence even if nominally "invalid". This is how the HTML5 spec works and it's been phenomenally successful.
  • ekidd 1y ago
    For security reasons, the correct answer on how process invalid UTF-8 is (and needs to be) "throw away the data like it's radioactive, and return an error." Otherwise you leave yourself wide open to validation bypass attacks at many layers of your stack.
    • modeless 1y ago
      This is only true because the interpretation is not defined, so different implementations do different things.
      • cryptonector 1y ago
        That's not true. You're just not allowed to interpret them as characters.
    • account42 1y ago
      This is rarely the correct thing to do. Users don't particularly like it if you refuse to process a document because it has an error somewhere in there. Even for identifiers you probably want to do all kinds of normalization even beyond the level of UTF-8 so things like overlong sequences and other errors are really not an inherent security issue.
  • moefh 1y ago
    > This is how the HTML5 spec works and it's been phenomenally successful. Unicode does have a completely defined way to interpret invalid UTF-8 byte sequences by replacing them with the U+FFFD ("replacement character"). You'll see it used (for example) in browsers all the time. Mandating acceptance for every invalid input works well for HTML because it's meant to be consumed (primarily) by humans. It's not done for UTF-8 because in some situations it's much more useful to detect and report errors instead of making an automatic correction that can't be automatically detected after the fact.
• kevincox 1y ago
  > Every ASCII encoded file is a valid UTF-8 file. More importantly, that file has the same meaning. Same with the converse.
• Androth 1y ago
  meh. it's a brilliant design to put a bandage over a bad design. if a language can't fit into 255 glyphs, it should be reinvented.
  • rmunn 1y ago
    Sun Tzu would like a word or two with you.
• ofou 1y ago
  UTF-8 should be a universal tokenizer
• wrp 1y ago
  I need to call out a myth about UTF-8. Tools built to assume UTF-8 are not backwards compatible with ASCII. An encoding INCLUDES but also EXCLUDES. When a tool is set to use UTF-8, it will process an ASCII stream, but it will not filter out non-ASCII. I still use some tools that assume ASCII input. For many years now, Linux tools have been removing the ability to specify default ASCII, leaving UTF-8 as the only relevant choice. This has caused me extra work, because if the data processing chain goes through these tools, I have to manually inspect the data for non-ASCII noise that has been introduced. I mostly use those older tools on Windows now, because most Windows tools still allow you to set default ASCII.
  • account42 1y ago
    Do you have an actual example where this causes an issue? "ASCII" tools mostly just passed along non-ASCII bytes unchanged even before UTF-8.
  • kccqzy 1y ago
    That's not a myth about UTF-8. That's a decision by tools not to support pure ASCII.
  • int_19h 1y ago
    The usual statement isn't that UTF-8 is backwards compatible with ASCII (it's obvious that any 8-bit encoding wouldn't be; that's why we have UTF-7!). It's that UTF-8 is backwards compatible with tools that are 8-bit clean.
    • wrp 1y ago
      Yes, the myth I was pointing out is based on loose terminology. It needs to be made clear that "backwards compatible" means that UTF-8 based tools can receive but are not constrained to emit valid ASCII. I see a lot of comments implying that UTF-8 can interact with an ASCII ecosystem without causing problems. Even worse, it seems most Linux developers believe there is no longer a need to provide a default ASCII setting if they have UTF-8.
• deleted 1y ago
  [deleted]
• jrochkind1 1y ago
  It really is, in so many ways. It is amazing how successful it's been.
• smoyer 1y ago
  Uvarint also has the property of a file containing only ascii characters still being a valid ascii file.
• lyu07282 1y ago
  UTF-8 was a huge improvement for sure, but I was, 20-25 years ago, working with LATIN-1 (so 8 bit charcters) which was a struggle in the years it took for everything to switch to UTF-8, the compatibility with ASCII meant you only really notice something was wrong when the data had special characters not representable in ASCII but valid LATIN-1. So perhaps breaking backwards compatibility would've resulted in less data corruption overall.
• digianarchist 1y ago
  I read online that codepoints are formatted with 4 hex chars for historical reasons. U+41 (Latin A) is formatted as U+0041.
• Mikhail_Edoshin 1y ago
  I once saw a good byte encoding for Unicode: 7 bit for data, 1 for continuation/stop. This gives 21 bit for data, which is enough for the whole range. ASCII compatible, at most 3 bytes per character. Very simple: the description is sufficient to implement it.
  • rmunn 1y ago
    Probably a good idea, but when UTF-8 was designed the Unicode committee had not yet made the mistake of limiting the character range to 21 bits. (Going into why it's a mistake would make this comment longer than it's worth, so I'll only expound on it if anyone asks me to). And at this point it would be a bad idea to switch away from the format that is now, finally, used in over 99% of all documents online. The gain would be small (not zero, but small) and the cost would be immense.
    • int_19h 1y ago
      Didn't they limit the range to 21 bits because UTF-16 has that limitation?
      • rmunn 1y ago
        That is indeed why they limited it, but that was a mistake. I want to call UTF-16 a mistake all on its own, but since it predated UTF-8, I can't entirely do so. But limiting the Unicode range to only what's allowed in UTF-16 was shortsighted. They should, instead, have allowed UTF-8 to continue to address 31 bits, and if the standard grew past 21 bits, then UTF-16 would be deprecated. (Going into depth would take an essay, and at this point nobody cares about hearing it, so I'll refrain).
        • gpvos 1y ago
          I suppose it's still possible to extend to 31 bits in the future, once UTF-16 has become obsolete enough. How big is the need for it right now?
• dmz73 1y ago
  UTF8 is a horrible design. The only reason it was widely adopted was backwards compatibility with ASCII. There are large number of invalid byte combinations that have to be discarded. Parsing forward is complex even before taking invalid byte combinations in account and parsing backwards is even worse. Compare that to UTF16 where parsing forward and backwards are simpler than UTF8 and if there is invalid surrogate combination, one can assume it is valid UCS2 char.
  • moefh 1y ago
    UTF-16 is an abomination. It's only easy to parse because it's artificially limited to 1 or 2 code units. It's an ugly hack that requires reserving 2048 code points ("surrogates") from the Unicode table just for the encoding itself. It's also the reason why Unicode has a limit of about 1.1 million code points: without UTF-16, we could have over 2 billion (which is the UTF-8 limit).
• Andrex 1y ago
  What are the perceived benefits of UTF-16 and 32 and why did they come about? I could ask Gemini but HN seems more knowledgeable.
  • peterfirefly 1y ago
    UTF-16 is a hack that was invented when it became clear that UCS-2 wasn't gonna work (65536 codepoints was not enough for everybody). Almost the entire world could have ignored it if not for Microsoft making the wrong choice with Windows NT and then stubbornly insisting that their wrong choice was indeed correct for a couple of decades. There was a long phase where some parts of Windows understood (and maybe generated) UTF-16 and others only UCS-2.
    • kccqzy 1y ago
      Besides Microsoft, plenty of others thought UTF-16 to be a good idea. The Haskell Text type used to be based on UTF-16; it only switched to UTF-8 a few years ago. Java still uses UTF-16, but with an ad hoc optimization called CompactStrings to use ISO-8859-1 where possible.
      • peterfirefly 1y ago
        A lot of them did it because they had to have a Windows version and had to interface with Windows APIs and Windows programs that only spoke UTF-16 (or UCS-2 or some unspecified hybrid). Java's mistake seems to have been independent and it seems mainly to have been motivated by the mistaken idea that it was necessary to index directly into strings. That would have been deprecated fast if Windows had been UTF-8 friendly and very fast if it had been UTF-16 hostile. We can always dream.
        • int_19h 1y ago
          There are many other examples, and while some of them are derived from the ones you give, others are independent. JavaScript is an obvious one, but there's also e.g. Qt and NSString in Objective-C, ICU etc. There really was a time when UTF-16 (or rather UCS2) made sense.
• Mikhail_Edoshin 1y ago
  One aspect of Unicode that is probably not obvious is that with Unicode it is possible to keep using old encodings just fine. You can always get their Unicode equivalents, this is what Unicode was about. Otherwise just keep the data as is, tagged with the encoding. This nicely extends to filesystem "encodings" too.
  • Mikhail_Edoshin 1y ago
    For example, modern Python internally uses three forms (Latin-1, UTF-16 and 32) depending on the contents of the string. But this can be done for all encodings and also for things like file names that do not follow Unicode. The Unicode standard does not dictate everything must take the same form; it can be used to keep existing forms but make them compatible.
• vismit2000 1y ago
  Karpathy's "Let's build the GPT Tokenizer" also contains a good introduction to Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32 in the first 20 minutes: https://www.youtube.com/watch?v=zduSFxRajkE https://www.youtube.com/watch?v=zduSFxRajkE
• gritzko 1y ago
  I specialize in protocol design, unfortunately. A while ago I had to code some Unicode conversion routines from scratch and I must say I absolutely admire UTF-8. Unicode per se is a dumpster fire, likely because of objective reasons. Dealing with multiple Unicode encodings is a minefield. I even made an angry write-up back then https://web.archive.org/web/20231001011301/http://replicated.cc/concepts/unicode https://web.archive.org/web/20231001011301/http://replicated... UTF-8 made it all relatively neat back in the day. There are still ways to throw a wrench into the gears. For example, how do you handle UTF-8 encoded surrogate pairs? But at least one can filter that out as suspicious/malicious behavior.
  • sedatk 1y ago
    > For example, how do you handle UTF-8 encoded surrogate pairs? Surrogate pairs aren’t applicable to UTF-8. That part of Unicode block is just invalid for UTF-8 and should be treated as such (parsing error or as invalid characters etc).
    • gritzko 1y ago
      In theory, yes. In practice, there are throngs of parsers and converters who might handle such cases differently. https://seriot.ch/projects/parsing_json.html https://seriot.ch/projects/parsing_json.html
      • sedatk 1y ago
        I mean hopefully not, but the linked example is about JSON parsing, not UTF-8.
        • gritzko 1y ago
          A big chunk of bugs there are Unicode related, that is my point. When people parse JSON they don't think that they also parse Unicode.
  • cryptonector 1y ago
    > Unicode per se is a dumpster fire Maybe as to emojis, but otherwise, no, Unicode is not a dumpster fire. Unicode is elegant, and all the things that people complain about in Unicode are actually problems in human scripts.
• betimsl 1y ago
  The story is that Ken and Rob were at a diner when Ken gave structure to it and wrote the initial encode/decode functions on napkins. UTF-8 is so simple yet it required a complex mind to do it.
• drpixie 1y ago
  UTF-8 is a neat way of encoding 1M+ code points in 8 bit bytes, and including 7 bit ASCII. If only unicode were as neat - sigh. I guess it's way too late to flip unicode versions and start again avoiding the weirdness.
• akoboldfrying 1y ago
  I take it you could choose to encode a code point using a larger number of bytes than are actually needed? E.g., you could encode "A" using 1, 2, 3 or 4 bytes? Because if so: I don't really like that. It would mean that "equal sequence of code points" does not imply "equal sequence of encoded bytes" (the converse continues to hold, of course), while offering no advantage that I can see.
• vismit2000 1y ago
  UTF-8 Everywhere Manifesto: https://utf8everywhere.org/ https://utf8everywhere.org/
• incanus77 1y ago
  I had fun building a little UTF-8 playground about a year ago to drive some of the concepts home and to go back to as a cheatsheet. https://github.com/incanus/learn-utf8 https://github.com/incanus/learn-utf8
  • vishnuharidas 1y ago
    I made an interactive one since I couldn't find anything that allows individually set/unset bits and see what happens. Here: https://utf8-playground.netlify.app/ https://utf8-playground.netlify.app/
    • incanus77 1y ago
      Very nice! That's fun to play with.
• dolmen 1y ago
  Anyone remembers what UTF7.5 or UTF7,5 was? I can't find references to its description(s)...
  • dolmen 1y ago
    Finally found a description here: http://www.czyborra.com/utf/ http://www.czyborra.com/utf/
• kunley 1y ago
  In case no one mentioned that yet in this thread, here's the story how it was invented by Ken Thompson & Rob Pike over a dinner. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://doc.cat-v.org/bell_labs/utf-8_history https://doc.cat-v.org/bell_labs/utf-8_history
• xkcd1963 1y ago
  What I find inconvenient about emoji characters is the differential length counting in programming languages
  • kccqzy 1y ago
    That's a problem with programming languages having inconsistent definitions of length. They could be like Swift where the programmer has control over what counts as length one. Or they could decide that the problem shouldn't be solved by the language but by libraries like ICU.
• sjapkee 1y ago
  Until you interact with it as a programmer
• nottorp 1y ago
  Hmm i count at most 21 bits. Just 2 billion code points. Is that all Unicode can do? How are they going to fit all the emojis in?
  • danhau 1y ago
    The max code point in Unicode is 0x10FFFF. ceil(log2(0x10FFFF+1)) = 21. So yes, a Unicode codepoint requires only 21 bits. 297334 codepoints have been assigned so far, that‘s about 1/4 of the available range, if my napkin math is right. Plenty of room for more emoji.
• max23_ 1y ago
  Good read, thank you! > Show the character represented by the remaiing 7 bits on the screen. I notice there is a typo.
  • vishnuharidas 1y ago
    Fixed that, thank you!
• transfire 1y ago
  So brilliant that we’re all still using ASCII!† † With an occasional UNICODE flourish.
• gnufx 1y ago
  It's worth noting that Stallman had earlier proposed a design for Emacs "to handle all the world's alphabets and word signs" with similar requirements to UTF-8. That was the etc/CHARACTERS file in Emacs 18.59 (1990). The eventual international support implemented in Emacs 20's MULE was based on ISO-2022, which was a reasonable choice at the time, based on earlier Japanese work. (There was actually enough space in the MULE encoding to add UTF-8, but the implementation was always going to be inefficient with the number of bytes at the top of the code space.) Edit: see https://raw.githubusercontent.com/tsutsui/emacs-18.59-netbsd/refs/heads/master/etc/CHARACTERS https://raw.githubusercontent.com/tsutsui/emacs-18.59-netbsd...
• saltserv 1y ago
  [dead]
• danso 1y ago
  Love reading explorations of structures and technical phenomena that are basically the digital equivalent of oxygen in their ubiquity and in how we take them for granted
• xd1936 1y ago
  Excellent Computerphile video with Tom Scott about this as well: https://youtu.be/MijmeoH9LT4 https://youtu.be/MijmeoH9LT4
• senfiaj 1y ago
  UTF-8 is a nice extension for ASCII from the compatibility point of view, but it might be not the most compact especially if the text is not English like. Also, the variable character length makes it inconvenient to work with strings unless they are parsed/saved into/from 2/4 byte char array.
• z_open 1y ago
  kill Unicode. Done with this after these 25 byte single characters.
• curtisszmania 1y ago
  [dead]
• frollogaston 1y ago
  Seems obvious, ASCII had an unused bit, so you use it. Why did they even bother with UTF-16 and -32 then?
  • int_19h 1y ago
    Because the original design assumed that 16 bits are enough to encode everything worth encoding, hence UCS2 (not UTF-16, yet) being the easiest and most straightforward way to represent things.
    • frollogaston 1y ago
      Ah ok. Well even then, you end up spending 16 bits for every ASCII character.
• andig 1y ago
  So why wasn‘t 10 chosen as the 2-bytes prefix to gain 1 bit? Easier to detect encoding errors?