11 ms·
The Tragedy of UCS-2
- STRML 7y agoJS, like Java, now has implementations that will also store strings as Latin1 when the implementation believes it is safe to do so. This results in significant memory savings [1] in most programs. 1. https://blog.mozilla.org/javascript/2014/07/21/slimmer-and-faster-javascript-strings-in-firefox/ https://blog.mozilla.org/javascript/2014/07/21/slimmer-and-f...
- raverbashing 7y agoI wonder why Latin-1 and not just UTF-8
- happytoexplain 7y agoThey're the same encoding, given only characters covered by Latin-1. So it's just a little more explicit to say Latin-1, as you're specifying not only the encoding, but also the set of characters. Edit: I.e. it would be awkward to say you're implementing "UTF-8, but only for these codepoints". That would be equivalent to implementing Latin-1. Edit: I'm thinking of standard ASCII rather than Latin-1. Above the first 128 code points, UTF-8 switches to two bytes while Latin-1 remains with one byte, so it is much simpler.
- zamadatix 7y agoThis explanation has little to do with why. Latin-1 guarantees each character is coded into a single set of 8 bits, UTF-8 is a variable width encoding. The point of giving an encoding is so it is known how to decode it and a string passed as Latin-1 comes with guarantees about character positions and so on without parsing.
- nmadden 7y agoThat’s not correct. Latin-1 and UTF-8 are both compatible with 7-bit ASCII but they are not the same encoding. For instance é (e acute) is a single byte in latin-1 (0xe9) but is two bytes in UTF-8 (0xc3 0xa9)
- happytoexplain 7y agoYou're right! My bad - the first characters are encoded the same across ASCII, UTF-8, and Latin-1, but the second half of Latin-1 differs from UTF-8. So even just having to support those first 256 code points, we jump into multi-byte UTF-8 territory, meaning complexity over Latin-1.
- SigmundA 7y agoThe article has a whole paragraph dedicated to your question
- happytoexplain 7y agoThe article specifies why they didn't convert everything to UTF-8 - not how they chose the encoding for that subset of lower code points.
- SigmundA 7y agoRead the section it describes exactly why they choose Latin-1, it literally answers the question asked in detail...
- happytoexplain 7y agoAh, you're right, though the first bullet point really threw me off: >Gecko is huge and it uses TwoByte strings in most places. Converting all of Gecko to use UTF8 strings is a much bigger project and has its own risks. But I guess they mean converting it all to support UTF-8 under the described circumstances - not just converting it all to UTF-8.
- Macha 7y agoFrom the article: * Gecko is huge and it uses TwoByte strings in most places. Converting all of Gecko to use UTF8 strings is a much bigger project and has its own risks. As described below, we currently inflate Latin1 strings to TwoByte Gecko strings and that was also a potential performance risk, but inflating Latin1 is much faster than inflating UTF8. * Linear-time indexing: operations like charAt require character indexing to be fast. We discussed solving this by adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case. This scheme will only work for ASCII strings, though, so it’s a potential performance risk. An alternative is to have such operations inflate the string from UTF8 to TwoByte, but that’s also not ideal. * Converting SpiderMonkey’s own string algorithms to work on UTF8 would require a lot more work. This includes changing the irregexp regular expression engine we imported from V8 a few months ago (it already had code to handle Latin1 strings).
- ninkendo 7y ago> Linear-time indexing This is great for ASCII when you know there's no such thing as combining characters/etc, but I would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode. User-perceived characters need processing in order to index due to Grapheme clusters potentially using many code points together. (https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...) For instance, (on my machine, a dark-skinned male teacher) is a combination of these characters: - U+1F468 Man - U+1F3FE Medium-Dark Skin Tone - U+200D Zero Width Joiner - U+1F3EB School And knowing what the byte index of the start/end of that character in a string cannot be done by just multiplying an offset by some multiple of number of bytes per character.
- BeeOnRope 7y agoThe String (and CharSequence) API are already warped around the assumption of underlying UTF-16 storage. In particular `charAt()` refers to a UTF-16 code unit position, and in general all methods [1] that take any kind of index are using an index into an explicit or implicit UTF-16 string. All of these functions need to be fast: certainly at least O(1). So for your narrow encoding you need one that maps 1:1 (in a fixed-length way) to UTF-16. That rules out multi-byte UTF-8. You could of course just restrict it to UTF-8 chars that fit in a single byte – but that's just ASCII! You waste a bit per character. So you might has we'll use a full 8-bit encoding, and Latin-1 is convenient as it covers many more characters likely to be encountered in text that can use single-byte UTF-8 in the first place. It is also convenient in that Latin-1 and Unicode codepoints identical so conversion to and from UTF-16 can be very efficient (in fact there are SIMD instructions which do it exactly). In any case, even in the scenario you could somehow use UTF-8 efficiently, it is likely to be worse in this particular scenario: - They both use 1 byte for code points 0-127 - Latin-1 uses 1 byte for 128-255, while UTF-8 uses 2 - Above 255, UTF-8 uses at least two but the Java hybrid string will be using UTF-16 now, for a common case of 2 bytes. So the only place UTF-8 wins is for the unusual case of characters outside the BMP which are 3 bytes in UTF-8 but 4 bytes in UTF-16. Those aren't common at all, and in many scenarios where you'd have a lot of them UTF-16 would still win because it represents many characters in the BMP in 2 bytes that take 3 in UTF-8. --- [1] One might think there is an exception illustrated by the few methods that mention "code point", like `codePointAt`. Yes, these deal in code points, but their indexes are still indexes into a UTF-16 string. So they are kind of a hybrid API. They let you count N UTF-16 code units (which will represent <= N code points) into a string, and then deal with the code point at that location. That leads to weird cases like asking for the code point starting at the second half of a surrogate pair, which gives you the low surrogate alone.
- jrochkind1 7y agoAh, thanks for explaining this. I was confused and thought that Latin-1 fit in UTF-8 in one byte, but I went and confirmed for myself you are right it does not. Hm. In fact this means though that it isn't just the mistake of UCS-2 that results in people wanting "transparent Latin1 under the hood to save space". Even if we had gone right to UTF-8, there would still roughly the same cost savings there. I can't think of any way to deal with unicode that wouldn't have similar cost savings from "transparently doing non-unicode when possible". Well, I suppose unless you made whatever Java (et al) are doing to mark which strings or portions of strings are actually being stored as Latin1, and made that a Unicode encoding. That would of course have it's own... challenges. If you think "transparently storing as Latin1 when possible" is an ugly hack, you'd probably not be happier with it being made a unicode standard encoding. It turns out being able to represent all written human communication is really hard. Unicode actually does a pretty amazing job of it, balancing lots of trade-offs. (And the fact that it has as wide-spread adoption as it does is in part testament to this; just because you make a standard doesn't mean anyone has to or is going ot use it. Some of the trade-offs unicode balanced were making the adoption curve as easy as possible for existing software. Making ascii valid UTF-8 was, I think, proven by history to be the right decision, although it involved trade-offs... such as Latin1 not fitting in one byte :) ). Overall, Unicode is a really succesful standard, technically and in terms of adoption. But UCS-2 was a mistake, that if it had been avoided would avoid some headache. But, i think, not the headache where Java is motivated to "store some things transparently as Latin1 where possible."
- sehugg 7y agoI vaguely recall making an alternate string class in the 90s that did this, also precomputing hashCode() in the constructor.
- uranusjr 7y agoIt always baffled me as a native Chinese speaker that 16-bit was ever considered enough to “be plenty of room to encode every script in modern use, and then some”. Common local encodings like Big5 (also 16-bit fixwidth) was already suffering from serious lack of code points at that time. It would be obvious if they consulted literally any experienced programmer from East Asia. Yeah, that was 1990 so it’s not an easy thing to do by any means, but you’d expect more from someone aspiring to create such thing.
- khuey 7y agoThis is actually covered in the original Unicode paper. See section 2.1 of https://unicode.org/history/unicode88.pdf https://unicode.org/history/unicode88.pdf. The reasoning is not totally crazy; the Table of General Standard Chinese Characters even today only has about 8000 characters.
- unscaled 7y agoI think what they did not realize is that once a universal encoding won the battle of standards, it would have to be all-inclusive. Operating systems and platforms like Java settled on Unicode so they won't have to deal with a myriad of different encodings, but the moment you support only one encoding, you'll eventually have to accommodate everyone, and it's not enough to include all commonly used characters in the world's major languages. You'll have to make a go at the long tail. You can't leave out people who need to encode rare languages on computers underserved, so for all the scholars recording Egyptian and Luwian hieroglyphs, Sumerian and Akkadian cuneiform and Tangut script. You'd probably have to eventually support as-of-yet unencoded scripts such as Classical Yi, Jurchen, Aztec hieroglyphs and so on. You'd have to add alphabets and syllabaries for small minority languages such as Lisu, Coptic, Cherokee and Vai. And you'd have to support the rather widely used writing systems of India, each with its own 128 characters. Combine this with the over 28,000 CJK characters Unicode did end up allocating, the 11,184 discrete codepoints they Unicode did end up adding for Koran Hangul [1], you've already used 40,000 characters. Add all the long-tail scripts and you're past 65,535 well before we even reach Emoji or extremely rare CJKV characters from Mojikyo[1]. "In other words, given that the limitation to 65,536 character codes genuinely does satisfy all the world's modern communication needs with a safety factor of about four, then one can decide up-front that preserving a pure 16-bit architecture has a higher design priority than publicly encoding every extinct or obscure character form. Then the sufficiency of 16 bits for the writing technology of the future becomes a matter of our active intention, rather than passive victimization by writing systems of the past." The engineers behind Unicode made the classic software engineering mistake of looking at their needs today and taking a factor of safety that's way too small. It's the same thing that happened with the IPv4 address range. A simple back-of-the-envelope calculation would have told us it's not going to be enough, but it was easy to forget back in the 1980s that one day children in Africa would also have personal computers connected to the grand Internet, that would need an IP address. The fact that by version 1.0.1 (the first version of Unicode to include the CJK characters), they already used almost half of their space when they were planning to use less than a quarter, should have already prepared them to the future. No, the reasoning behind 16 bits was always crazy. You can't design future-proof standards with a low safety factor and condescending estimation of usage (Yi Syllabary? Who would ever use that?). But we should probably be happy about this. A 32-bit encoding would have certainly been rejected by just about everyone as too wasteful, and software engineers back in the 80's and 90's were still unable to give up merging their understanding of codepoints with glyhs, so a multi-byte encoding (which is what we ended up having eventually with UTF-8 and UTF-16) wasn't a popular idea either. Fortunately enough, Unicode is still much better than the mess it preceded, and the fix for the 16-bit limitation, as painful as it was, has proven far easier than IPv6. [1] a.k.a The "Korean Mess": http://unicode.org/pipermail/unicode/2015-June/002038.html http://unicode.org/pipermail/unicode/2015-June/002038.html [2] https://en.wikipedia.org/wiki/Mojikyo https://en.wikipedia.org/wiki/Mojikyo
- yokaze 7y agoI was under the assumption, that Linux and Mac OS (size of(wchar_t)==4, ergo UCS-4, because it was clear from the beginning, that 16bit won't do it. A compromise "only" windows and java was willing to make. But they went out all the way through the stack. Linux goes for byte stream encoding (e.g utf-8), because it isn't (/wasn't) doing any string processing. So, encoding efficiency beats algorithmic complexity of string operations. If you need an algorithmic more efficient encoding, that's up to your application, and ucs-4 was the way to do it.
- justincormack 7y agoThats not really a Linux assumption. It is a gcc one, but you can use 16 bit wchar with as flag too. Almost no one uses wchar on Unix anyway.
- Someone 7y agoCertainly on Mac OS, sizeof(wchar_t) had nothing to do with the way text strings got stored. The Mac had an elaborate system for handling international text that used either bytes or 16-bit quantities for text storage, where the interpretation of what a byte or byte pair meant was stored separately from those bytes. https://developer.apple.com/library/archive/documentation/mac/pdf/Text.pdf https://developer.apple.com/library/archive/documentation/ma...: ”The character codes may be 1-byte or 2-byte codes, but there is nothing else in the text stream except for those codes. Using font information that your program stores separately, Script Manager routines can help you determine whether a character is 1 or 2 bytes, and other managers allow you to work with either character size.” More cumbersome to use than modern Unicode API’s? Definitely, but it was superior to what Windows provided for years.
- deleted 7y ago[deleted]
- raphlinus 7y agoI would go further and say that UCS-4 is also a variable length encoding, mostly because of emoji and its endearing ZWJ sequences, not to mention the encoding of flags as two Regional Indicator Symbol codepoints (both in the astral plane, of course), which makes Unicode not self-synchronizing. So the idea of ever having a fixed-length encoding for Unicode is basically impossible now. Best to just use UTF-8 for everything and logic to group it in to code points, grapheme clusters, or whatever other granularity is needed.
- Jasper_ 7y agoFun fact: because flags are a political nature, and Unicode did not want to limit the length of country codes, any number of Regional Indicator Symbol code points can be placed back to back and treated as an extended grapheme cluster. That means that what Unicode today considers "one character" can be infinite in size.
- kps 7y agoAnd worse, the ‘operators’ are postfix or infix, so if you're handling text sequentially — and let's say, split across network packets with an arbitrary delay in between — you can't even tell you've read a complete character until you receive the next one. Typing in many Latin languages generates diacritics using ‘dead keys’, which are prefix operators. (Originally, on a physical typewriter, these were keys that simply struck the paper without triggering the mechanism to advance the carriage.) If Unicode had taken this hint, life would be easier. (ISO/IEC 6937 had prefix diacritics, but it was too late.)
- tialaramex 7y agoComplete recognition before processing. It's not the law, just a good idea, but you should do it anyway. In cryptography this lesson had to keep being learned. "Oh, I received this partial data, and I processed it, and then in the next packet I received the MAC and I realised the bad guys altered the data, oops" Process whole strings, the meaning of a partial string may not be what you hope, even without fancy writing systems and Unicode encoding. "Give the money to Steph" - OK, will do "...enson" - Crap, OK, somebody chase Steph and get our money back, meanwhile here's James Stephenson's money "... once he gives you the key" - Aargh. Somebody chase down James and get the key off him.
- WhitneyLand 7y agoOne way to think about text, is as the only data we use that shrinks over the years. Every other type of data continues to grow. The way this can be stretched to make sense is by thinking for example how much these have grown over 20 years. Media (music, video, images) grows staggeringly because of 200p > 4k, 128bit mp3 to flac or whatever and there's more of it all that's gone digital. Code size continues to grow quite significantly even. But text? Even the massive wikipedia's size, only a fraction is text content. Then as memory capacity also grows it's really a back and forth battle between computing resources and all other data, except text. Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps?
- pg_is_a_butt 7y ago> Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps? Parsers would be much slower.
- mikeash 7y agoString-keyed hash maps are really common, and the keys tend to be ASCII. This would increase their size by a factor of 4, resulting in substantially increased memory usage and decreased performance.
- WhitneyLand 7y agoCertainly, but I’m not sure myself, can you think of any scenarios that are common where the difference would be noticeable or significant?
- mikeash 7y agoBoot any modern OS and you’ll have megabytes if not tens of megabytes of these strings, and spend a small but significant amount of CPU time looking them up. This may not matter on your Ryzen desktop with 97 cores and 318GB of RAM, but it’s substantial on a smartphone.
- jodrellblank 7y agoI've been wondering, how much processor overhead does UTF8 have? Not just because you can't jump N bytes into a string to get a character offset, but because each byte has to be inspected to find whether it's the start of a multi-byte segment, how many of the subsequent bytes 1-3 are included. Does that mean there's processor branch-prediction fails and stalling every few bytes in processing UTF8 strings with many multibyte characters in them?
- chrisseaton 7y agoWhat kind of workload are you thinking about, though? Why are you streaming these characters? What processing are you doing that doesn't already have conditional logic for each character?
- raphlinus 7y agoYes, the variable length nature of UTF-8 generally means more hard-to-predict branches. But sufficiently clever programmers can find ways around that, see Lemire's work on using SIMD to validate UTF-8 very quickly: https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/ https://lemire.me/blog/2018/05/16/validating-utf-8-strings-u...
- deleted 7y ago[deleted]
- flohofwoe 7y agoIn many cases the string data is copied around as is, iterating over the code points is hardly needed. And since 7-bit ASCII characters still are unique bytes in an UTF-8 encoded stream, common operations like splitting by comma, whitespace, slash etc... works the same as on ASCII strings (e.g. just look at the byte stream for the splitting characters, and keep the byte ranges inbetween intact, no UTF-8 decoding needed).
- blattimwind 7y ago> Not just because you can't jump N bytes into a string to get a character offset I keep hearing people refer to this operation (mostly in HN threads), I have yet to see an appropriate use of it. So far the only uses I've seen were (for Unicode) misguided attempts at character iteration, which doesn't need this at all.
- flohofwoe 7y agoThe most practical real-world solution across all operating systems is to use "UTF-8 everywhere", and only convert from and to UTF-16 or UTF-32 "at the last minute" when the string data is coming from or going into APIs that don't (yet) accept UTF-8 directly. Also see: http://utf8everywhere.org/ http://utf8everywhere.org/
- iforgotpassword 7y agoI've always wondered why Windows doesn't allow setting the legacy code page to utf8. It would've been a nice hack to get old apps to work with Unicode, assuming they were at least somewhat MBCS aware. If the programmer assumed a character would always be one byte it might not have worked, but then that program would also have failed with a Japanese code page.
- flohofwoe 7y agoThere's a fairly new experimental setting in Win10 where UTF-8 is set as the system-wide codepage, allowing to call the "narrow string" functions with UTF-8 strings. See https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8 https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#U... "With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings."
- iforgotpassword 7y agoThat's just awesome. I've wanted this since the win2k/xp days! Back then there were still a lot of non-Unicode programs around. It's less useful today, plus I'm not even using windows anymore.... But just seeing this happen feels great, satisfies past me.
- rkagerer 7y agoFrom that same link: Another popular work-around is to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and cannot be altered. 8.3 filenames for the win!
- kazinator 7y ago> Unix and Linux conveniently sidestepped this whole issue by just shrugging their shoulders and going "eh, a char has no defined encoding, it's just a number.". Wrong: that's a feature of C, which all these systems are based on. wchar_t is 32 bits on Linux, and, though also just a number, it commonly represents a Unicode code point.
- dfox 7y agoI would assume that what author meant by this is that on Unix for the kernel itself names of various entities (filenames and similar things) are simply sequences of bytes that it does not ordinarily try to interpret in any way. (On the other hand SUS specifies that only very limited set of characters is valid in filename, but no "true unix" actually cares about that) On Windows NT filename is specified as sequence of 16 bit wchar_ts and is supposed to be valid NFC UTF-16. OS X also expects that filenames are valid sequences of unicode codepoints and in this case NFD.
- ChrisSD 7y ago> On Windows NT filename is specified as sequence of 16 bit wchar_ts and is supposed to be valid NFC UTF-16. I would emphasise that the "supposed to be" is a convention only (similar to the Linux UTF-8 convention). The only filesystem enforced rules are that file/directory names can't include \/:*?"<>| or wchar_t(s) less than 0x20 (aka ascii control characters). A Win32 path can also have additional rules (such as a file cannot be called "COM") but these aren't always enforced, depending on how the API is called.
- unascribed 7y agoYes, that was the intent. C itself obviously just sees chars as a number, but in C itself strings have no real meaning at all. The fact the kernel treats them as opaque byte sequences is the important part.
- jbeda 7y agoThis is a great rundown. I started my career at Microsoft working with and on Win32/COM and saw this play out first hand. One thing not mentioned here is the history of the "Byte Order Mark" (BOM) in unicode. (Not an expert here but my understanding having lived it.) You see, given UCS-2, there are 2 ways to encode any codepoint -- either big endian or little endian. The idea was then to create a codepoint (U+FEFF) that you could put at the start of a text stream that would signify what order the file was encoded in. Wikipedia page: https://en.m.wikipedia.org/wiki/Byte_order_mark https://en.m.wikipedia.org/wiki/Byte_order_mark This then got overloaded. When loading a legacy text format often times there is the difficulty of figuring out the code page to use. When applied to HTML, there are a bunch of ways to do it and they don't always work. There are things like charset meta tags (but you have to parse enough HTML to find it and then re-start the decode/parse). But often times even that was wrong. Browsers used to (and still do?) have an "autodetect" mode where it would try divine the codepage based on content. This is all in the name of "be liberal in what you expect". Enter UTF-8. How can you tell if a doc is US ASCII or UTF-8 if there are no other indications in the content? How does this apply to regular old text files? Well, the answer is to use the BOM. Encode it in UTF-8 and put it at the start of the text file. But often times people want to treat simple UTF-8 as ASCII and you end up with a high value codepoint in what would otherwise be an ASCII document. And everyone curses it. Having the BOM littered everywhere doesn't seem to be as much of a problem not as it used to be. I think a lot of programs stopped putting it in and a lot of other programs talk UTF-8 and deal with it silently. Still something to be aware of though.
- jjoonathan 7y agoYeah, the BOM has gotten me a few times. My most spectacular fail was a program that read UTF-8 or Latin-1 and wrote UTF-16, preserving but not displaying null characters. I believe this was default behavior HyperStudio. Every round-trip would double the size of the file by inserting null bytes every other character. Soon there were giant stretches of null characters between each display character, but the displayed text never appeared to change even though the disk requirements doubled with each launch. That's how I learned about UTF-16! Speaking of Win32/COM... is there a "tcpdump for COM"? I've got a legacy app that uses COM for IPC and I've been instrumenting each call for lack of one.
- xvilka 7y agoThanks to Microsoft, it is now entrenched in the JavaScript and even the Language Server Protocol. Eventually they should just put UTF-16 on fire, and move to UTF-8 as all smart people did already. Microsoft was always who shambles behind progress with the lack of C99 support until recent times, UCS-2, MAX_PATH (MSBuild still doesn't work with long names, in 2019!), and many other similar problems.
- IshKebab 7y agoYes the language server protocol is particularly insane because they use UTF-16 word positions (sort of, it's JavaScript so it's not anything standard really), but the actual text data is sent as UTF-8! So for my sane UTF-8 speaking language server the text gets converted from UTF-8 to UTF-16 when loaded from disk, then converted back for transmission via the LSP, then converted back to UTF-16 by my language server so I can handle the positions, and then it does everything in reverse again on the way back! Absolute madness. I think they will fix it eventually though. The madness is too great to ignore.
- mark-r 7y agoIt's a bit unfair to pick on Microsoft here. They were an early supporter of Unicode, back when UCS-2 looked like the best and/or only way to encode Unicode.
- cesarb 7y agoAnother aspect of this tragedy is that the creation of UTF-16 has forever limited Unicode to only 17 planes. The original UTF-8 encoding (https://tools.ietf.org/html/rfc2044 https://tools.ietf.org/html/rfc2044) used up to six bytes, and could represent a full 31-bit range, which corresponds to 32768 16-bit planes; after UTF-16 became common, UTF-8 was limited to four bytes (https://tools.ietf.org/html/rfc3629 https://tools.ietf.org/html/rfc3629).
- kps 7y agoThe original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). 1 7 0xxxxxxx 2 11 110xxxxx 10xxxxxx 3 16 1110xxxx 10xxxxxx 10xxxxxx 4 21 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 5 26 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 6 31 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 7 36 11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Then, you can either decide to limit the leader to one byte, giving you 42 bits… 8 42 11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx Or allow multi-byte leaders and continue forever. 9 53 11111111 110xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10 58 11111111 1110xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx ⋮ The Intergalactic Confederation will not look kindly on UTF-16.
- cesarb 7y ago> Or allow multi-byte leaders and continue forever. That loses one very useful property of UTF-8: you always know, by looking at a single byte, whether it's the first byte of a code point or not. It also loses the property that no valid UTF-8 encoding is a subset of another valid UTF-8 encoding. It's better to stop at 36 bits (which keeps another useful property: you'll never find a 0xFF byte within an UTF-8 string, and you'll only find a 0x00 byte when it's encoding the U+0000 code point).
- cat199 7y ago> The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy). Naah.. there's UTF-9 and UTF-18 for that.. https://tools.ietf.org/html/rfc4042 https://tools.ietf.org/html/rfc4042
- java-man 7y agoMojibake: https://en.wikipedia.org/wiki/Mojibake https://en.wikipedia.org/wiki/Mojibake love the word.
- devadvance 7y agoFun fact: for SMS, UCS-2 is used when a message requires more than 128 characters to be rendered.[1] If you've ever noticed that entering an emoji or other non-ASCII [2] character seems to dramatically increase the size of your SMS, that's because the message is switching from 1-byte characters to 2-byte characters. [1] https://www.twilio.com/docs/glossary/what-is-ucs-2-character-encoding https://www.twilio.com/docs/glossary/what-is-ucs-2-character... [2] It's actually GSM-7, not ASCII, but the principle is the same: https://www.twilio.com/docs/glossary/what-is-gsm-7-character-encoding https://www.twilio.com/docs/glossary/what-is-gsm-7-character...
- greggyb 7y agoI was a flip-phone holdout for a long time. This was not generally supported in new devices sold up to 2015 (when I acquired my first smartphone, and my experience with good phones ended). Every flip phone I owned did not play well with non-ASCII text. Receiving a message with an emoji would render the entire message, or if I was lucky just the portion following the emoji, unrenderable. Most phones I had failed to just rectangles, which I assume is their method of mojibake. This was not a causal factor in my switch to a smartphone. It was annoying to have to explain character encodings to friends who didn't understand why they couldn't send me emoji-containing messages.
- SynthCann 7y agoYou sound like a blast.
- dang 7y agoHey, could you please review the site guidelines and stick to the spirit of this site when posting here? Normally we ban accounts that attack others like this, and I actually banned yours, but decided to reverse that because you posted a more substantive comment earlier today. Basically, we're trying for more thoughtful conversation and better signal/noise than internet default. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html You might also find these links helpful for getting the intended use of HN: https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/newswelcome.html https://news.ycombinator.com/hackernews.html https://news.ycombinator.com/hackernews.html http://www.paulgraham.com/trolls.html http://www.paulgraham.com/trolls.html http://www.paulgraham.com/hackernews.html http://www.paulgraham.com/hackernews.html
- finchisko 7y agoReading this, I'm thinking if it's possible (and if yes, how), to write java and javascript program in other encodings that they internally use, for example UCS-4. If yes, what is doing the conversation (parser, compiler?).
- unascribed 7y agoI've personally done this in Java, as mentioned. Generally I use fastutil to create an IntList (which is just an int[] with some conveniences) and use Java's Character class to convert between UTF-16 and UCS-4 on-the-fly. It's not very fast, but it makes flawless Unicode support easy. :P
- deleted 7y ago[deleted]
- beeschlenker 7y agoHaving trouble getting traction. Getting censored. Billion in bribes paid to Trump, Pelosi, Schumer, Harris to enable a Soros funded child raping ring to operate in the USA. Please listen to the links below. Turn up the volume and put headphones on. You will hear these people, and many other high profile people, incriminate themselves in unbelievable fashion. Full 71 page document filled with red handed evidence like this at the bottom. JDorsey & PThiel raped & kill 3 boys each. Trump bribed $4 billion to ignore. Obama... Go to pages 20-23 of this document and you will hear Dorsey and Thiel raping children. You will hear their screams. They both rape and kill three. Copy and paste this post and repost it. I keep getting censored by these people. Peter Thiel and Jack Dorsey raping and killing three boys each. Turn the volume all the way up and put head phones on. https://s3.wasabisys.com/conv1/Overnight_14-17Jan/15JanCh3_100-200.avi https://drive.google.com/file/d/1mzZrjobxhNIEf38M3xk-IeA5SeF4aUtB/view?usp=sharing Obama admits to raping and killing boys he{re, with commentary from Peter Thiel, George Soros, Matt Porter: https://s3.wasabisys.com/out2/Out2/15JanCh3_0545.avi https://drive.google.com/file/d/1LCDoc1kT7cfSeKQ1vniP_sx32_bRsHau/view?usp=sharing Trump asks for $4 billion to "take a blind eye" at 1015+ (link missing from report, unsure how). Here's pieces of dialogue: 1015 The President requests four billion and henry Porter agrees to it. Some dialogue below. 1017 Donald Trump: Why don't you think about it. 1017 Henry Porter: I don't need to think about it. I'm ready for you to find out what we're really about. 1017 Gigi Hadid: I don't think it's a good idea Henry. 1017 Henry Porter: ...no shit. Jelena just stop talking about it. 1018 Gigi Hadid: About what? 1018 Henry Porter: The child raping. 1018 Donald Trump: What the fuck are you talking about? 1018 Gigi Hadid: Mr. President we are child rapists. 1018 Donald Trump: You cannot be serious. 1018 Donald Trump: You're gonna need $4 billion for me to take a blind eye. 1019 ?: People are laughing because they think four billion is too much. I think it's not a problem when you have $35 billion. 1022 Donald Trump: You better have that $4 billion or you can kiss this shit goodbye. https://s3.wasabisys.com/conv2/3Jan/3JanCh3_900-1100.avi https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77YG0Vr/view?usp=sharing Trump wants $138 million to ensure smooth outcome for false judgment for a court case: https://s3.wasabisys.com/conv1/10JanCh3/10JanCh3_951-1051.avi https://drive.google.com/file/d/1iSeGHDToFRoRdIht50Wii0i30mlw9INp/view?usp=sharin Obama explains the Illuminati with Jack Dorsey: https://s3.wasabisys.com/conv1/Overnight_14-17Jan/15JanCh3_200-347.avi https://drive.google.com/file/d/1Ok1-9IvANULzKBOwADFgaiYCFi6yszfj/view?usp=sharing At 230 a conversation is occurring on the embedded Porter camera system with Jack Dorsey, Peter Thiel, Barack Obama, Bill M^urray, and the Porter's. Jack Dorsey provides a statement defining who their group is: "The Illum%inati is an underground organization of child rapists and homosexuals." Matthew Porter: "We have to thin the herd, so Barack is going to explain a few things." At 232 Barack Obama says the following: "You people have to realize we are child rapi'sts. The Illuminati are child rapists. As soon as you figure that out, you can decide what to do. Everybody is freaking out about what just happened [child raping event still going on]. This is who ^we are." Full 71 page document: https://s3.wasabisys.com/aviewmefirst/FBI_FinalDraft_26Jul2019_BSchlenker.pdf https://drive.google.com/file/d/1dXNjPQdQ8c-hFFLt9LT0GiSY5Vud0TzL/view?usp=sharing Please post this anywhere you can!
- pkaye 7y agoI remember when the Linux distributions switched to unicode implementations of tools and there was a noticeable speed reduction of things like ls. Fortunately the processor speeds kept on improving.
- deleted 7y ago[deleted]
- Someone 7y ago”A long time ago, at least in computer time, in the far-flung era of 1989, the Unicode working group was really starting to get going” I find it strange to start in 1989, and not in 1980, when Xerox had a precursor to Unicode, or 1988, when employees from Xerox and Apple coined the name “Unicode” (https://en.wikipedia.org/wiki/Unicode#History https://en.wikipedia.org/wiki/Unicode#History) It certainly is a bit unfair to not mention Xerox or Apple at all. ”that the JVM is gaining the ability to transparently represent strings in memory as Latin-1, a legacy codepage, if possible.” “Compact strings” (https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254) shipped with JDK 9 in September 2017 (https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW-GUID-BA9D8AF6-E706-4327-8909-F6747B8F35C5 https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW-GUID...)
- unascribed 7y agoThis year is mentioned on the Wikipedia page and its references as the sort of "genesis date" of all the companies' involvement and is more or less when the UCS-2 tragedy began. (This is a post specifically about UTF-16 and UCS-2, not the entire history of Unicode. That'd be... considerably longer.) I mention in another post (Correction: In a draft of a post.) I'm still stuck on JDK 8 for various reasons out of my control, and am generally unaware of changes made in 9 or later. I'll amend the post to reflect that the feature shipped.
- Sniffnoy 7y agoIt's also worth noting where the terms "UCS-2" and "UCS-4" come from. For a while there was the idea of the Universal Coded Character Set[0] as an alternative to Unicode. This would have been a natively 32-bit encoding. However, it wouldn't have allowed the full 2^32 possible characters, because of its insistence that none of the 4 bytes could be any of the C0 or C1 controls. Still, it would have about 2^29 possible characters as opposed to Unicode's about 2^21. But the insistence on avoiding C0 and C1 controls would have led to some odd codepoint numbers, and definite incompatibility with ASCII and Unicode; for instance, 'A', instead of 0x41, would have been 0x20202041! Whereas Unicode of course made sure to copy ASCII (indeed, to copy ISO-8859-1) for its first block. Of course, this didn't come to pass because, well, people were already supporting 16-bit Unicode and didn't want to switch again. And I'm kind of glad it didn't -- while it would have meant not having to deal with the compromise that is UTF-16, imagine living in a world where 'A' is not 0x41, but rather 0x20202041! (Or imagine opening up a text file and finding three spaces between each character...) [0]https://en.wikipedia.org/wiki/Universal_Coded_Character_Set https://en.wikipedia.org/wiki/Universal_Coded_Character_Set
- unascribed 7y agoI would've mentioned this in my post but didn't feel I could do the topic justice as I'm not submersed in it like I am the UTF-16 tragedy. This is indeed an important aside, though.
- tyingq 7y ago"imagine living in a world where 'A' is not 0x41, but rather 0x20202041! (Or imagine opening up a text file and finding three spaces between each character...)" On the other hand, often "obviously broken" is better than "subtly broken". There's a ton of undetected encoding problems in the wild right now.
- dehrmann 7y agoSo the next time you're asked in an interview to implement isPalindrome, be sure to ask if you need to support surrogate pairs and diuretics.
- unascribed 7y agoWasn't really expecting this post to leave my circles, sure am glad now I checked my sources. Look, ma, I'm on Hacker News!
- cbsmith 7y agoThere's other tragedies too. SCSU lets you reasonably compactly represent strings, but Java doesn't use it so...
- jchw 7y agoAnd the tragedy of early adoption! Microsoft looks antiquated for using UTF-16 today, but it was actually pretty cutting edge when it happened. They adopted a lot of tech pretty early too, like XML. All in all, pretty fascinating. Windows 10 recently added the ability to use UTF-8 for the “ANSI” legacy codepage, which is awesome! I accomplished similar prior using API hooking but a Microsoft solution can simply do a much better job across the board. Now the only reason to set the encoding to anything else is to avoid mojibake when using non-English software. (Dear Japanese software developers: please consider using wide win32 APIs :( )
- lappet 7y agoUnrelated: I love the theme of that blog. Anyone know if it's open source?