4 ms·

I would disagree with the statement that UTF-16 gained popularity. UTF-16 is mostly used where it provided an easy upgrade path from the 16 bit restricted UCS-

by bhaak 8y ago

I would disagree with the statement that UTF-16 gained popularity. UTF-16 is mostly used where it provided an easy upgrade path from the 16 bit restricted UCS-2 to enabling full Unicode coverage. Switching from UCS-2 to UTF-8 would have been a pain either because of dropping backward compatibility or ridiculous wrappers to get it working. Microsoft was one of the first companies to implement Unicode, so they didn't have much of a choice. Java could probably have switched to UTF-8 before 1.0 was released but I guess the supposed runtime overhead for not being able to access characters at indexes was feared too much. Of course, they lost that anyway with UTF-16.

pwagland 8y ago
Java 1.0 was made back in the days when "65K characters would be enough for anyone!" hence they chose UCS-2. Java only supported UTF-16 as of Java 5. I suspect that if they could turn back time, they would have chosen UTF-8 from the get-go, but, as you say, that was a battle lost long ago.
- jillesvangurp 8y ago
  Java 9 released last year changed the internal string representation to not be UTF-16 anymore Up until then characters took at least 2 bytes in Java. Of course the internal representation and what is written to IO streams are two things. UTF-8 has been supported pretty much from day 1 along with a whole range of legacy encodings that used to be common. Using UTF-8 instead of UTF-16 internally is mostly a memory optimization and does not actually break any apis that I'm aware off (I'm sure there are corner cases, but am not aware of any issues here). Using UTF-16 was always somewhat valid in non western scripts where the most used characters would be represented using two bytes instead of just 1 like for most western scripts. But you are right it was a bit of a regrettable choice. If you consider that, e.g. asians mix western script with their local script, it is a lot less clear cut. Throw things like html in the mix and you are looking at a lot of ascii with bits of asian script here and there.

by bhaak 8y ago

pwagland 8y ago

Java 1.0 was made back in the days when "65K characters would be enough for anyone!" hence they chose UCS-2. Java only supported UTF-16 as of Java 5. I suspect that if they could turn back time, they would have chosen UTF-8 from the get-go, but, as you say, that was a battle lost long ago.

jillesvangurp 8y ago
Java 9 released last year changed the internal string representation to not be UTF-16 anymore Up until then characters took at least 2 bytes in Java. Of course the internal representation and what is written to IO streams are two things. UTF-8 has been supported pretty much from day 1 along with a whole range of legacy encodings that used to be common. Using UTF-8 instead of UTF-16 internally is mostly a memory optimization and does not actually break any apis that I'm aware off (I'm sure there are corner cases, but am not aware of any issues here). Using UTF-16 was always somewhat valid in non western scripts where the most used characters would be represented using two bytes instead of just 1 like for most western scripts. But you are right it was a bit of a regrettable choice. If you consider that, e.g. asians mix western script with their local script, it is a lot less clear cut. Throw things like html in the mix and you are looking at a lot of ascii with bits of asian script here and there.