4 ms·

This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal

by SiVal 11y ago

This sounds like a ridiculous idea. Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS). This proposal doesn't challenge that claim, nor could it. Java strings are sequences of Unicode code points, logically at least. This proposal is just about how to encode such a sequence internally. There are numerous alternatives for encoding sequences of Unicode code points, each with its own pros and cons. UTF-8, UTF-16, UTF-32, and various compression schemes emphasize backward compatibility, or space savings, or rapid, random access of characters, or whatever. Different designs optimized for different benefits, with different associated costs. The UCS character sequence is the universal thing, but how it is represented should be optimized for the application. It sounds as though this proposal is only about minimizing the number of bytes needed to represent a sequence of UCS characters, but ISO 8859-1 was never designed as a minimal-space representation of UCS characters. Its design reflects entirely different goals. If all you care about is an internal (not externally visible) representation that minimizes string size, you should do a proper statistical analysis first. You will find, for example, that the curly quotes and dashes that are nearly ubiquitous in serious English text and the Euro character so fundamental to international economics are far more commonly included in Java strings than are most control characters. Yet control characters are a part of ISO 8859-1, and those vastly more useful characters are not. Why, if your goal is to optimize for space, would you privilege so many obscure control characters in your default, internal representation, yet force a switch to a different encoding for any string on a blog containing an m-dash or on an e-commerce site containing a Euro sign? Instead, if it's all about space, and it is an internal, hidden representation, you do a proper statistical analysis of exactly what it is you most need to represent, and either assign bytes in inverse proportion to commonness, or you use a proper information-theoretic compression "encoding" scheme based on the results of your analysis and your weighted goals. Or you stick with the existing, standard, run-anywhere representation Java has used since the beginning. Either way, switching to ISO 8859-1 for this makes no sense.

Skinney 11y ago
Firefox did this recently, with great success. It's a simple solution that will reduce memory usage (and this, garbage collection). It makes perfect sense.
- SiVal 11y ago
  HTML is ISO 8859-1 by default, which privileges that legacy encoding within conformant browsers and web servers. That's the reason for all the "character entity" stuff. The same is not true of Java.
  - justincormack 11y ago
    No longer true, HTML5 is UTF8 by default.
  - gsnedders 11y ago
    HTML is a locale specific encoding by default. (No, really.) Most locales default to windows-1252, not ISO-8859-1. That said, all conforming HTML documents must now use UTF-8.
pilif 11y ago
> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually uses UTF-16 encoding to encode characters outside of the BMP, but none of the APIs actually know about this and still assume UCS-2. This leads to the API being wrong about character lengths and to regular expressions wrongly matching certain strings and sometimes destroying data when you apply them to replace substrings. Case in point is this little test class here: https://gist.github.com/5745601abbfbf7068fcd https://gist.github.com/5745601abbfbf7068fcd which prints 2 on the console. I've also given a talk about this at a swiss JS conference in 2012: http://pilif.me/unicode.pdf http://pilif.me/unicode.pdf - JS has/had the same issues as Java in that regard. The only well-known contemporary languages that get this (somewhat) right are Perl (since forever), Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you.
- Hinrik 11y ago
  > The only well-known contemporary languages that get this (somewhat) right are Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. And Perl.
  - pilif 11y ago
    You are right of course (I knew this as you can see when you look at the PDF I linked - slide 50). I have edited my original post.
- aardvark179 11y ago
  > unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. Not entirely true. There are codePoint*, indexOf(int ch), and various other methods on String which do work correctly in code points. Unfortunately old methods can't be changed without affecting compatibility.
jdscriven 11y ago
ISO 8859-1 does have the unique property that each of its 256 code points map directly to the first 256 Unicode code points.
cesarb 11y ago
> Either way, switching to ISO 8859-1 for this makes no sense. ISO 8859-1 has an important property: it maps directly to the first 256 Unicode codepoints (by construction, since Unicode was designed so that property was true). This means that converting from ISO 8859-1 to UCS-2 or UTF-16 is a simple zero-extension. That has direct performance benefits when doing any operation which mixes ISO 8859-1 and UCS-2 or UTF-16, which will happen often if the internal representation for String can use either. For instance, searching for an ASCII substring within a long UTF-16 string (ASCII maps directly to the first 128 ISO 8859-1 values, another important property). Yes, it's not the most minimal representation, but since Strings are so common in Java, performance is important. And a lot of Strings use only the first 256 codepoints. It's mentioned in the linked article: "Data gathered from many different applications indicates that strings are a major component of heap usage and, moreover, that most String objects contain only Latin-1 characters."
david-given 11y ago
...but, if I've understood the proposal correctly, the public interface isn't changing. The user of String will never know, nor care, whether their string is implemented as UTF-8 or ISO-8859-1. This is purely an implementation detail. Personally, I'd be astonished if JVMs didn't already do this: way back when when I was working for a company that did embedded JVMs, it was one of the first optimisations we did. When class files were loaded, we'd look at each string, and if they were ISO-8859-1 we'd use a byte-based representation rather than a word-based one. (Programmatically created strings were always words, of course.) Even in an environment where most of our customers were Asian the savings were huge. Java uses a lot of identifier strings.
karmakaze 11y ago
Why is this even being discussed? It's only an implementation optimization, no change to the language interface. I don't recall such a fuss around compressed pointers. This is so much better than the number of software stacks that introduce ByteString types specifically for this benefit. If String just did this when it could, then it could just be knowingly used in place of each roll your own.

by SiVal 11y ago

Skinney 11y ago

Firefox did this recently, with great success. It's a simple solution that will reduce memory usage (and this, garbage collection). It makes perfect sense.

SiVal 11y ago
HTML is ISO 8859-1 by default, which privileges that legacy encoding within conformant browsers and web servers. That's the reason for all the "character entity" stuff. The same is not true of Java.
- justincormack 11y ago
  No longer true, HTML5 is UTF8 by default.
- gsnedders 11y ago
  HTML is a locale specific encoding by default. (No, really.) Most locales default to windows-1252, not ISO-8859-1. That said, all conforming HTML documents must now use UTF-8.

pilif 11y ago

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually uses UTF-16 encoding to encode characters outside of the BMP, but none of the APIs actually know about this and still assume UCS-2. This leads to the API being wrong about character lengths and to regular expressions wrongly matching certain strings and sometimes destroying data when you apply them to replace substrings. Case in point is this little test class here: https://gist.github.com/5745601abbfbf7068fcd https://gist.github.com/5745601abbfbf7068fcd which prints 2 on the console. I've also given a talk about this at a swiss JS conference in 2012: http://pilif.me/unicode.pdf http://pilif.me/unicode.pdf - JS has/had the same issues as Java in that regard. The only well-known contemporary languages that get this (somewhat) right are Perl (since forever), Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you.

Hinrik 11y ago
> The only well-known contemporary languages that get this (somewhat) right are Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. And Perl.
- pilif 11y ago
  You are right of course (I knew this as you can see when you look at the PDF I linked - slide 50). I have edited my original post.
aardvark179 11y ago
> unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. Not entirely true. There are codePoint*, indexOf(int ch), and various other methods on String which do work correctly in code points. Unfortunately old methods can't be changed without affecting compatibility.

jdscriven 11y ago

ISO 8859-1 does have the unique property that each of its 256 code points map directly to the first 256 Unicode code points.

cesarb 11y ago

> Either way, switching to ISO 8859-1 for this makes no sense. ISO 8859-1 has an important property: it maps directly to the first 256 Unicode codepoints (by construction, since Unicode was designed so that property was true). This means that converting from ISO 8859-1 to UCS-2 or UTF-16 is a simple zero-extension. That has direct performance benefits when doing any operation which mixes ISO 8859-1 and UCS-2 or UTF-16, which will happen often if the internal representation for String can use either. For instance, searching for an ASCII substring within a long UTF-16 string (ASCII maps directly to the first 128 ISO 8859-1 values, another important property). Yes, it's not the most minimal representation, but since Strings are so common in Java, performance is important. And a lot of Strings use only the first 256 codepoints. It's mentioned in the linked article: "Data gathered from many different applications indicates that strings are a major component of heap usage and, moreover, that most String objects contain only Latin-1 characters."

david-given 11y ago

...but, if I've understood the proposal correctly, the public interface isn't changing. The user of String will never know, nor care, whether their string is implemented as UTF-8 or ISO-8859-1. This is purely an implementation detail. Personally, I'd be astonished if JVMs didn't already do this: way back when when I was working for a company that did embedded JVMs, it was one of the first optimisations we did. When class files were loaded, we'd look at each string, and if they were ISO-8859-1 we'd use a byte-based representation rather than a word-based one. (Programmatically created strings were always words, of course.) Even in an environment where most of our customers were Asian the savings were huge. Java uses a lot of identifier strings.

karmakaze 11y ago

Why is this even being discussed? It's only an implementation optimization, no change to the language interface. I don't recall such a fuss around compressed pointers. This is so much better than the number of software stacks that introduce ByteString types specifically for this benefit. If String just did this when it could, then it could just be knowingly used in place of each roll your own.