9 ms·

UTF-8 is one of the most brilliant things I've ever seen. I only wish it had been invented and caught on before so many influential bodies started using UCS-2

• SethMLarson 5y ago
  100% agree, it's really rare that there's a ~blanket solution to a whole class of problems. "Just use UTF-8!"
• josephg 5y ago
  Absolutely. At least it’s well supported now in very old languages (like C) and very new languages (like Rust). But Java, Javascript, C# and others will probably be stuck using UCS-2 forever.
  • stewx 5y ago
    What is stopping people from encoding their Java, JS, and C# files in UTF-8?
    • maskros 5y ago
      Nothing, but Java's "char" type is always going to be 16-bit.
      • josephg 5y ago
        Yep. In javascript (and Java and C# from memory) the String.length property is based on the encoding length in UTF16. It’s essentially useless. I don’t know if I’ve ever seen a valid use for the javascript String.length field in a program which handles Unicode correctly. There’s 3 valid (and useful) ways to measure a string depending on context: - Number of Unicode characters (useful in collaborative editing) - Byte length when encoded (these days usually in utf8) - and the number of rendered grapheme clusters All of these measures are identical in ASCII text - which is an endless source of bugs. Sadly these languages give you a deceptively useless .length property and make you go fishing when you want to make your code correct.
        • account42 5y ago
          > and the number of rendered grapheme clusters This is also rarely useful unless you are working with a monospace font where all grapheme clusters have the same width, which is probably none if you support double-width characters. More likely what you are interested in is the display length with a particular font or column count with a monospace font.
      • tialaramex 5y ago
        Java's char is a strong competitor for most stupid "char" type award. I would give it to Java outright if not for the fact that C's char type doesn't define how big it is at all, nor whether it is signed. In practice it's probably a byte, but you aren't actually promised that, and even if it is a byte you aren't promised whether this byte is treated as signed or unsigned, that's implementation dependant. Completely useless. For years I thought char was just pointless, and even today I would still say that a high level language like Java (or Javascript) should not offer a "char" type because the problems you're solving with these languages are so unlikely to make effective use of such a type as to make it far from essential. Just have a string type, and provide methods acting on strings, forget "char". But Rust did show me that a strongly typed systems language might actually have some use for a distinct type here (Rust's char really does only hold the 21-bit Unicode Scalar Values, you can't put arbitrary 32-bit values in it, nor UTF-16's surrogate code points) so I'll give it that.
        • mark-r 5y ago
          The only guarantee that C gives you is that sizeof char == 1, and even that's not as useful as it looks.
          • SAI_Peregrinus 5y ago
            It also guarantees that char is at least 8 bits.
            • account42 5y ago
              And POSIX does guarantees that CHAR_BIT == 8 so in practice this is only a concern on embedded platform where you are only dealing with "C-ish" anyway.
              • SAI_Peregrinus 5y ago
                Or any other non-POSIX system. I'm not sure what Windows guarantees, but it's definitely not a POSIX system.
    • jasode 5y ago
      >What is stopping [...] Java, JS, and C# files in UTF-8? The output of files on disk can be UTF-8. The continued use of UCS-2 (later revised to UTF16) is happening in the runtime because things like the Win32 API which C# uses is UCS-2. The internal raw memory of layout of strings in Win32 is UCS-2. *EDIT to add correction
      • colejohnson66 5y ago
        UTF-16*, not UCS-2. Although there are probably many programs that assume UCS-2.
        • nwallin 5y ago
          Note that the asterisk in `UTF-16*` is a really big asterisk. I fixed a UCS-16 bug last week at my day job.
          • WorldMaker 5y ago
            Yeah, there's sometimes a lot more hacks like WTF-8 and WTF-16 in practice on UCS-2 originally systems (including Windows and JS) than is healthy: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
        • mark-r 5y ago
          When Windows adopted Unicode, I think the only encoding available was UCS-2. They converted pretty quickly to UTF-16 though, and I think the same is true of everybody else who started with UCS-2. Unfortunately UTF-16 has its own set of hassles.
          • account42 5y ago
            Technically, they converted to WTF-16 [0] since many places, including filenames, allow you to use unpaired surrogates. [0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
      • kevin_thibedeau 5y ago
    • mark-r 5y ago
      Nothing at all, and in fact there's a site set up specifically to advocate for this: https://utf8everywhere.org/ https://utf8everywhere.org/ The biggest problem is when you're working in an ecosystem that uses a different encoding and you're forced to convert back and forth constantly. I like the way Python 3 does it - every string is Unicode, and you don't know or care what encoding it is using internally in memory. It's only when you read or write to a file that you need to care about encoding, and the default has slowly been converging on UTF-8.
      • account42 5y ago
        The problem with "every string is Unicode" is if you want to represent things that look like unicode but aren't really guaranteed to be unicode. This includes filenames on Windows (WTF-18 aka arbitrary WCHAR sequences) and Linux (arbitrary byte sequences) that are interpreted as UTF-16 / UTF-8 for display purposes but limiting yourself to valid UTF-16 / UTF-8 means that you cannot represent all paths that you might come across.
  • HideousKojima 5y ago
    There's actually a proposal with a decent amount of support to add utf-8 strings to C#. Probably won't be added to the language for another 3 or 4 years (if ever) but it's not outside the realm of possibility. Edit: The proposal for anyone interested https://github.com/dotnet/runtime/issues/933 https://github.com/dotnet/runtime/issues/933
• BiteCode_dev 5y ago
  Like anything new, people had a hard time with it at the beginning. I remember that I got a home assignment in an interview for a PHP job. The person evaluating my code said I should not have used UTF8, which causes "compatibility problems". At the time, I didn't know better, and I answered that no, it was explicitly created to solve compatibility problems, and that they just didn't understand how to deal with encoding properly. Needing-less to say, I didn't get the job :) Same with Python 2 code. So many people, when migrating to Python 3, suddenly though python 3 encoding management was broken, since it was raising so many UnicodeDecodingError. Only much later people realize the huge number of programs that couldn't deal with non ASCII characters in file paths, html attributes or user names, because they just implicitly assume ASCII. "My code used to work fine", they said. But it worked fine on their machine, set to an english locale, tested only using ascii plain text files on their ascii named directories with their ascii last name.
  • andrepd 5y ago
    That's in general a problem with dynamic languages with weak type systems. How "Your code runs without crashing" is really really != "your code works". How do people even manage production python! A bug could be lurking anywhere, undetected until it's actually run. Whereas in a compiled language with a strong type system, "your code compiles" is much closer to "your code is correct".
    • deleted 5y ago
      [deleted]
    • digisign 5y ago
      There are a number of mitigations, so those kind of bugs are quite rare. In our large code base, about 98% of bugs we find are of the "we need to handle another case" variety. Pyflakes quickly finds typos which eliminates most of the rest.
      • wk_end 5y ago
        This is the difference between people who embrace static typing and everyone else. A static type lover hears that 98% of your bugs are of the "we need to handle another case" variety and says, "well, that means you could have gotten rid of 98% of your bugs with better typing".
• junon 5y ago
  There are (a few) very good reasons not to use UTF-8. It's a great encoding but not suitable for all cases. For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8. But when performance isn't critical, utf-8 should be the default. I don't see a reason for any other encoding.
  • jfk13 5y ago
    > For example, constant time subscripting, or improved length calculations, are made possible by encodings other than utf-8. Assuming you mean different encoding forms of Unicode (rather than entirely different and far less comprehensive character sets, such as ASCII or Latin-1), there are very few use cases where "subscripting" or "length calculations" would benefit significantly from using a different encoding form, because it is rare that individual Unicode code points are the most appropriate units to work with. (If you're happy to sacrifice support for most of the world's writing systems in favour of raw performance for a limited subset of scripts and text operations, that's different.)
  • ninkendo 5y ago
    Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. If you're hoping that a fixed offset gives you a user-percieved character boundary, then you're not handling composed characters or zero-width-joiners or any number of other things that may cause a grapheme cluster to be composed of multiple UTF code points. The "fixed" size of code points in encodings like UTF-32 are just that: code points. Whether a code point corresponds with anything useful, like the boundary of a visible character, will always require linear-time indexing of the string, in any encoding. (*) Approximately nothing. If you're in a position where you've somehow already vetted that the text is of a subset of human languages where you're guaranteed to never have grapheme clusters that occupy more than a single code point, then you maybe have a use case for this, but I'd argue you really just have a bunch of bugs waiting to happen.
    • irq-1 5y ago
      > Constant time subscripting is a myth. There's nothing(*) useful to be obtained by adding a fixed offset to the base of your string, in any unicode encoding, including UTF-32. What about UTF-256? Maybe not today, maybe not tomorrow, but someday...
• chrismorgan 5y ago
  The really tragic thing is that UTF-8 was invented before UTF-16. But a few big companies had put a couple of years of heavy investment into UCS-2, and weren’t willing to let that and the attendant pain they were beginning to foist on developers and users alike go to waste, and so ruined Unicode with UTF-16 and the disaster called surrogates that is the cause of the significant majority of programming languages handling strings incorrectly (e.g. JavaScript uses potentially ill-formed Unicode, indexed by UTF-16 code unit; and Python strings are sequences of code points rather than scalar values). If only they had reversed course and said “sorry for all the pain we were just starting to put you through, that fixed-width 16-bit encoding thing didn’t pan out, we’re going back to 8-bit encodings with this UTF-8 thing that is conveniently also backwards-compatible with ASCII”. If only.
  • account42 5y ago
    > the disaster called surrogates that is the cause of the significant majority of programming languages handling strings incorrectly It also wastes valuable code space that could be used to have more characters encode as 3 bytes instead of 4 in UTF-8.