5 ms·

I have always wondered why they didn't do that, it seems to be such a simple optimization. With UTF-8 you would give up some constant time operations like chara

I have always wondered why they didn't do that, it seems to be such a simple optimization. With UTF-8 you would give up some constant time operations like character lookups, while ISO-8859-1 shouldn't have any performance regressions I can think of. Because once you add a non-latin character to a String and the encoding has to change, the data has to be copied anyway due to String immutability.

chrisseaton 12y ago
I guess at the time the JVM was less aggressively optimised, and UCS-2 was a single simple approach that represented all required code points. Anything else would probably have looked like a micro-optimisation.
- agumonkey 12y ago
  Also fit well with World Wide Web + Write Once Run Anywhere : any OS in any locale.
- MrBuddyCasino 12y ago
  But back then we had less memory, and Java was never that memory efficient to begin with. I wouldn't say double-digit memory savings are in the realm of overly pedantic micro optimisations. One language whose name escapes me took an approach where the internal representation is always unicode codepoints, and Strings have methods for projecting those .toUTF8() or whatever to interact with the outside world - I liked that a lot due to its simplicity, anyone knows which language that was?
- desdiv 12y ago
  At the time UCS-2 was the best thing they could choose from. Remember, Java predates UTF-8.
pilif 12y ago
>With UTF-8 you would give up some constant time operations like character lookups if Java had a non-broken string API, even with the current UCS-2 based solution they couldn't do constant-time character lookups. And even if Java went the extra mile of using characters big enough to encode every code point of a string, constant lookup would still not be possible because users are often interested graphemes, not characters.
- MrBuddyCasino 12y ago
  That is sadly true, but at least for the Basic Multilingual Plane, charAt() and codePointAt() return correct results in constant time. The only non-broken methods are afaik codePointCount() and offsetByCodePoint(), which, as you pointed out, are not that interesting anyway. Do you know of an example of a well-designed String API? I'd be interested in what trade-offs other language designers made (newer ones, like Rust/Swift/Go/?).
  - ygra 12y ago
    IMHO Python gets strings very right. The internal representation is variable, depending on the string's content (to save memory), and the external API only gives you access to code points.
    - gsnedders 12y ago
      That's only Python 3.3 and above; before that it was an under-defined mess, in CPython depending on compile-time options and often broken with the same UCS-2/UTF-16 mess as numerous other languages.
    - TheLoneWolfling 12y ago
      The problem with Python strings is that innocuous operations can occasionally cause drastic memory usage ballooning. A simple string append can, for instance, require up to 5x the memory consumed by the string. Worse, this problem will only show up if someone ever enters in a high enough code point. Better hope any user-defined data never gets turned into part of a large string at any point, or you have a potential memory problem down the line.

chrisseaton 12y ago

I guess at the time the JVM was less aggressively optimised, and UCS-2 was a single simple approach that represented all required code points. Anything else would probably have looked like a micro-optimisation.

agumonkey 12y ago
Also fit well with World Wide Web + Write Once Run Anywhere : any OS in any locale.
MrBuddyCasino 12y ago
But back then we had less memory, and Java was never that memory efficient to begin with. I wouldn't say double-digit memory savings are in the realm of overly pedantic micro optimisations. One language whose name escapes me took an approach where the internal representation is always unicode codepoints, and Strings have methods for projecting those .toUTF8() or whatever to interact with the outside world - I liked that a lot due to its simplicity, anyone knows which language that was?
desdiv 12y ago
At the time UCS-2 was the best thing they could choose from. Remember, Java predates UTF-8.

pilif 12y ago

>With UTF-8 you would give up some constant time operations like character lookups if Java had a non-broken string API, even with the current UCS-2 based solution they couldn't do constant-time character lookups. And even if Java went the extra mile of using characters big enough to encode every code point of a string, constant lookup would still not be possible because users are often interested graphemes, not characters.

MrBuddyCasino 12y ago
That is sadly true, but at least for the Basic Multilingual Plane, charAt() and codePointAt() return correct results in constant time. The only non-broken methods are afaik codePointCount() and offsetByCodePoint(), which, as you pointed out, are not that interesting anyway. Do you know of an example of a well-designed String API? I'd be interested in what trade-offs other language designers made (newer ones, like Rust/Swift/Go/?).
- ygra 12y ago
  IMHO Python gets strings very right. The internal representation is variable, depending on the string's content (to save memory), and the external API only gives you access to code points.
  - gsnedders 12y ago
    That's only Python 3.3 and above; before that it was an under-defined mess, in CPython depending on compile-time options and often broken with the same UCS-2/UTF-16 mess as numerous other languages.
  - TheLoneWolfling 12y ago
    The problem with Python strings is that innocuous operations can occasionally cause drastic memory usage ballooning. A simple string append can, for instance, require up to 5x the memory consumed by the string. Worse, this problem will only show up if someone ever enters in a high enough code point. Better hope any user-defined data never gets turned into part of a large string at any point, or you have a potential memory problem down the line.