3 ms·
Using UTF-16 is a mistake because it doesn't represent anything well. Even if you use UTF-32 you still can't fit all characters in a single codepoint because so
by astrange 6y ago
Using UTF-16 is a mistake because it doesn't represent anything well. Even if you use UTF-32 you still can't fit all characters in a single codepoint because some of them (like flag emoji) are composed. So you should expect things to be variable length. But once you do, UTF-8 is more efficient than UTF-16 even if you're processing CJK text because it so often has ASCII characters mixed in.
Cocoa/ObjC has the same problem - actually Java is based on Cocoa so that's probably where it came from.
- nearbuy 6y agoEven with the surrogate pairs, Java's choice strikes a compromise between the memory efficiency of UTF-8 and the performance of fixed size code points. Most of the string methods work as if all characters are two bytes. So if, for example, you have a string composed entirely of 4 byte emojis, charAt(10) would return the first part of the 6th emoji, not the 11th emoji. For many cases, strings won't use any characters outside of UCS-2. In these cases, programmers gain the performance advantages of fixed sized characters without worrying about the downsides of surrogate pairs. It doesn't have the small memory footprint of UTF-8, but one could argue that if memory size is really important and fixed size characters are not, you're better off using in-memory compression for your strings, which will beat UTF-8. If you are using characters outside of UCS-2, then Java's string methods are a potential source of confusion and bugs. There's always a trade-off, right?
- discreteevent 6y agoFrom java9 onward if all the characters in a string are ASCII then java will store them in a byte array with one byte per character. This ends up covering a lot of strings.