4 ms·
[citation needed!] As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the
by mmastrac 17y ago
[citation needed!]
As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively.
To be honest, I've never actually had to use anything outside of the Basic Multilingual Plane. That's not from lack of breadth in my day-to-day job either - I wrote the first implementation of our web crawler/search engine at DotSpots and had to deal with fetching and indexing pages in languages from most of the BMP (mainly English, Russian and Chinese).
Anyways, I strongly disagree with your assertion that Java was "generally considered" to have "made a mistake" by using UTF-16. It's made my life far easier and the memory costs aren't an issue on today's machines. We don't live in an ASCII world most of the time and storing strings internally as UTF-8 completely ignores this fact.
[edit]
Just so it's clear: UTF-8 only wins when you are representing pure ASCII text. For everything else, it either breaks even or loses. For most Chinese text, UTF-8 is 50% larger:
For characters <= U+0080 (ie: ABC), you win by one byte.
For characters > U+0080 but <= U+07FF (ie: Ȁɐ), you break even at two bytes.
Everything higher than U+07FF in the BMP, < U+10000 (ie: 丂且⬄☃), you lose (by one byte).
For characters >= U+10000 you break even again at four bytes.
If you really want to optimize for ASCII text in Java, there's always byte[] and you're free to wrap a CharSequence around it.
- nradov 17y agoHave you actually worked with code points and surrogate pairs for the java.lang.String class? It's a huge hassle and doing it correctly requires at least triple the amount of code as doing it wrong. The problem is that earlier versions of Java used UCS-2 and so a single real-world character was always represented by a single Java char value. But then the Java maintainers later changed the string representation to UTF-16 so now a single real-world character could take up either one or multiple Java char values. A lot of new code is still being written which implicitly, and incorrectly, assumes that 1 character == 1 char. I even intentionally do it wrong sometimes myself because dealing with code points leads to such code bloat.
- mmastrac 17y agoI've worked with the codePointAt and various traversal methods, yes. They are a pain to get right and most Java code that uses substrings is broken in this regard. In general practice, however, this is a non-issue. Characters outside of the Basic Multilingual Plane are not in common use, especially on the web. It's not a perfect programming practice, but it's very pragmatic. Making everyone pay the development tax of variable-sized characters for any sort of multi-lingual code just means that more code will be written incorrectly.
- nradov 17y agoMy team has to write code that will accept and parse data from sources we don't even know about today, and continue working with minimal maintenance over a 15+ year life cycle. I don't think that's an uncommon situation. There's a lot more to software development than just toy web applications.
- Daniel_Newby 17y ago"In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively." This is a red herring. Because of combining characters, it is rarely valid to slice between Unicode code points, regardless of encoding. Even in the BMP, a semantic symbol can be composed of multiple code points.