5 ms·

What is stopping people from encoding their Java, JS, and C# files in UTF-8?

by stewx 5y ago

maskros 5y ago
Nothing, but Java's "char" type is always going to be 16-bit.
- josephg 5y ago
  Yep. In javascript (and Java and C# from memory) the String.length property is based on the encoding length in UTF16. It’s essentially useless. I don’t know if I’ve ever seen a valid use for the javascript String.length field in a program which handles Unicode correctly. There’s 3 valid (and useful) ways to measure a string depending on context: - Number of Unicode characters (useful in collaborative editing) - Byte length when encoded (these days usually in utf8) - and the number of rendered grapheme clusters All of these measures are identical in ASCII text - which is an endless source of bugs. Sadly these languages give you a deceptively useless .length property and make you go fishing when you want to make your code correct.
  - account42 5y ago
    > and the number of rendered grapheme clusters This is also rarely useful unless you are working with a monospace font where all grapheme clusters have the same width, which is probably none if you support double-width characters. More likely what you are interested in is the display length with a particular font or column count with a monospace font.
- tialaramex 5y ago
  Java's char is a strong competitor for most stupid "char" type award. I would give it to Java outright if not for the fact that C's char type doesn't define how big it is at all, nor whether it is signed. In practice it's probably a byte, but you aren't actually promised that, and even if it is a byte you aren't promised whether this byte is treated as signed or unsigned, that's implementation dependant. Completely useless. For years I thought char was just pointless, and even today I would still say that a high level language like Java (or Javascript) should not offer a "char" type because the problems you're solving with these languages are so unlikely to make effective use of such a type as to make it far from essential. Just have a string type, and provide methods acting on strings, forget "char". But Rust did show me that a strongly typed systems language might actually have some use for a distinct type here (Rust's char really does only hold the 21-bit Unicode Scalar Values, you can't put arbitrary 32-bit values in it, nor UTF-16's surrogate code points) so I'll give it that.
  - mark-r 5y ago
    The only guarantee that C gives you is that sizeof char == 1, and even that's not as useful as it looks.
    - SAI_Peregrinus 5y ago
      It also guarantees that char is at least 8 bits.
      - account42 5y ago
        And POSIX does guarantees that CHAR_BIT == 8 so in practice this is only a concern on embedded platform where you are only dealing with "C-ish" anyway.
        SAI_Peregrinus 5y ago
        Or any other non-POSIX system. I'm not sure what Windows guarantees, but it's definitely not a POSIX system.
        account42 5y ago
        How many non-embedded non-POSIX systems do you know? Windows also guarantees CHAR_BIT == 8 and since most software is first written for Windows or POSIX there is plenty of software that assumes that CHAR_BIT == 8. That means that anything that will want to run general software needs to also ensure CHAR_BIT == 8 - not to mention all the algorithms and data formats designed around you being able to efficiently access octets. The only platforms that can get away with CHAR_BIT != 8 are precisely those that have software specially written for them, i.e. embedded systems.
  - saagarjha 5y ago
    You’re being far too harsh. The Java char type isn’t “stupid”; really, it’s just unfortunate in hindsight. There are plenty of decisions that were stupid even at the time they were decided, and this isn’t that: people actually thought that 2 bytes was enough for all characters, and Han unification was going to work. Looking backward this is “obviously” futile but certainly not then. C’s character type, FWIW, has a use: it more or less indicates the granularity that is efficiently addressable by the host architecture. Trying to use it for more than that is generally not that fruitful, but it definitely has a purpose and it’s pretty good at that. Finally, speaking of unfortunate decisions, Rust happens to make one that I don’t particularly like: it lets you misalign characters (and panics), which is…not great. It would be much nicer if the view just don’t let you do this unless you specifically asked for bytes or something.
jasode 5y ago
>What is stopping [...] Java, JS, and C# files in UTF-8? The output of files on disk can be UTF-8. The continued use of UCS-2 (later revised to UTF16) is happening in the runtime because things like the Win32 API which C# uses is UCS-2. The internal raw memory of layout of strings in Win32 is UCS-2. *EDIT to add correction
- colejohnson66 5y ago
  UTF-16*, not UCS-2. Although there are probably many programs that assume UCS-2.
  - nwallin 5y ago
    Note that the asterisk in `UTF-16*` is a really big asterisk. I fixed a UCS-16 bug last week at my day job.
    - WorldMaker 5y ago
      Yeah, there's sometimes a lot more hacks like WTF-8 and WTF-16 in practice on UCS-2 originally systems (including Windows and JS) than is healthy: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
  - mark-r 5y ago
    When Windows adopted Unicode, I think the only encoding available was UCS-2. They converted pretty quickly to UTF-16 though, and I think the same is true of everybody else who started with UCS-2. Unfortunately UTF-16 has its own set of hassles.
    - account42 5y ago
      Technically, they converted to WTF-16 [0] since many places, including filenames, allow you to use unpaired surrogates. [0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- kevin_thibedeau 5y ago
mark-r 5y ago
Nothing at all, and in fact there's a site set up specifically to advocate for this: https://utf8everywhere.org/ https://utf8everywhere.org/ The biggest problem is when you're working in an ecosystem that uses a different encoding and you're forced to convert back and forth constantly. I like the way Python 3 does it - every string is Unicode, and you don't know or care what encoding it is using internally in memory. It's only when you read or write to a file that you need to care about encoding, and the default has slowly been converging on UTF-8.
- account42 5y ago
  The problem with "every string is Unicode" is if you want to represent things that look like unicode but aren't really guaranteed to be unicode. This includes filenames on Windows (WTF-18 aka arbitrary WCHAR sequences) and Linux (arbitrary byte sequences) that are interpreted as UTF-16 / UTF-8 for display purposes but limiting yourself to valid UTF-16 / UTF-8 means that you cannot represent all paths that you might come across.

What is stopping people from encoding their Java, JS, and C# files in UTF-8?

by stewx 5y ago

What is stopping people from encoding their Java, JS, and C# files in UTF-8?

maskros 5y ago

Nothing, but Java's "char" type is always going to be 16-bit.

josephg 5y ago
Yep. In javascript (and Java and C# from memory) the String.length property is based on the encoding length in UTF16. It’s essentially useless. I don’t know if I’ve ever seen a valid use for the javascript String.length field in a program which handles Unicode correctly. There’s 3 valid (and useful) ways to measure a string depending on context: - Number of Unicode characters (useful in collaborative editing) - Byte length when encoded (these days usually in utf8) - and the number of rendered grapheme clusters All of these measures are identical in ASCII text - which is an endless source of bugs. Sadly these languages give you a deceptively useless .length property and make you go fishing when you want to make your code correct.
- account42 5y ago
  > and the number of rendered grapheme clusters This is also rarely useful unless you are working with a monospace font where all grapheme clusters have the same width, which is probably none if you support double-width characters. More likely what you are interested in is the display length with a particular font or column count with a monospace font.
tialaramex 5y ago
Java's char is a strong competitor for most stupid "char" type award. I would give it to Java outright if not for the fact that C's char type doesn't define how big it is at all, nor whether it is signed. In practice it's probably a byte, but you aren't actually promised that, and even if it is a byte you aren't promised whether this byte is treated as signed or unsigned, that's implementation dependant. Completely useless. For years I thought char was just pointless, and even today I would still say that a high level language like Java (or Javascript) should not offer a "char" type because the problems you're solving with these languages are so unlikely to make effective use of such a type as to make it far from essential. Just have a string type, and provide methods acting on strings, forget "char". But Rust did show me that a strongly typed systems language might actually have some use for a distinct type here (Rust's char really does only hold the 21-bit Unicode Scalar Values, you can't put arbitrary 32-bit values in it, nor UTF-16's surrogate code points) so I'll give it that.
- mark-r 5y ago
  The only guarantee that C gives you is that sizeof char == 1, and even that's not as useful as it looks.
  - SAI_Peregrinus 5y ago
    It also guarantees that char is at least 8 bits.
    - account42 5y ago
      And POSIX does guarantees that CHAR_BIT == 8 so in practice this is only a concern on embedded platform where you are only dealing with "C-ish" anyway.
      - SAI_Peregrinus 5y ago
        Or any other non-POSIX system. I'm not sure what Windows guarantees, but it's definitely not a POSIX system.
        account42 5y ago
        How many non-embedded non-POSIX systems do you know? Windows also guarantees CHAR_BIT == 8 and since most software is first written for Windows or POSIX there is plenty of software that assumes that CHAR_BIT == 8. That means that anything that will want to run general software needs to also ensure CHAR_BIT == 8 - not to mention all the algorithms and data formats designed around you being able to efficiently access octets. The only platforms that can get away with CHAR_BIT != 8 are precisely those that have software specially written for them, i.e. embedded systems.
- saagarjha 5y ago
  You’re being far too harsh. The Java char type isn’t “stupid”; really, it’s just unfortunate in hindsight. There are plenty of decisions that were stupid even at the time they were decided, and this isn’t that: people actually thought that 2 bytes was enough for all characters, and Han unification was going to work. Looking backward this is “obviously” futile but certainly not then. C’s character type, FWIW, has a use: it more or less indicates the granularity that is efficiently addressable by the host architecture. Trying to use it for more than that is generally not that fruitful, but it definitely has a purpose and it’s pretty good at that. Finally, speaking of unfortunate decisions, Rust happens to make one that I don’t particularly like: it lets you misalign characters (and panics), which is…not great. It would be much nicer if the view just don’t let you do this unless you specifically asked for bytes or something.

jasode 5y ago

>What is stopping [...] Java, JS, and C# files in UTF-8? The output of files on disk can be UTF-8. The continued use of UCS-2 (later revised to UTF16) is happening in the runtime because things like the Win32 API which C# uses is UCS-2. The internal raw memory of layout of strings in Win32 is UCS-2. *EDIT to add correction

colejohnson66 5y ago
UTF-16*, not UCS-2. Although there are probably many programs that assume UCS-2.
- nwallin 5y ago
  Note that the asterisk in `UTF-16*` is a really big asterisk. I fixed a UCS-16 bug last week at my day job.
  - WorldMaker 5y ago
    Yeah, there's sometimes a lot more hacks like WTF-8 and WTF-16 in practice on UCS-2 originally systems (including Windows and JS) than is healthy: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- mark-r 5y ago
  When Windows adopted Unicode, I think the only encoding available was UCS-2. They converted pretty quickly to UTF-16 though, and I think the same is true of everybody else who started with UCS-2. Unfortunately UTF-16 has its own set of hassles.
  - account42 5y ago
    Technically, they converted to WTF-16 [0] since many places, including filenames, allow you to use unpaired surrogates. [0] https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
kevin_thibedeau 5y ago

mark-r 5y ago

Nothing at all, and in fact there's a site set up specifically to advocate for this: https://utf8everywhere.org/ https://utf8everywhere.org/ The biggest problem is when you're working in an ecosystem that uses a different encoding and you're forced to convert back and forth constantly. I like the way Python 3 does it - every string is Unicode, and you don't know or care what encoding it is using internally in memory. It's only when you read or write to a file that you need to care about encoding, and the default has slowly been converging on UTF-8.

account42 5y ago
The problem with "every string is Unicode" is if you want to represent things that look like unicode but aren't really guaranteed to be unicode. This includes filenames on Windows (WTF-18 aka arbitrary WCHAR sequences) and Linux (arbitrary byte sequences) that are interpreted as UTF-16 / UTF-8 for display purposes but limiting yourself to valid UTF-16 / UTF-8 means that you cannot represent all paths that you might come across.