7 ms·
JEP 400: UTF-8 by Default
- Animats 3y agoNice. Better late than never.
- morelisp 3y agoI guess I'll be happy to see this, but > Specify UTF-8 as the default charset without providing any means to change it — The compatibility impact of this change would be too high. Nah, nobody is getting this correct right now. You should've swung for the fences.
- invalidname 3y agoThis is pretty old news by now... But yes, it was long overdue.
- throwawaymobule 3y agoAnyone interested in going whole-hog with this, forking Java, and making char utf8 too? I'm guessing it's not worth it, even aside from legal costs.
- tialaramex 3y ago> making char utf8 too? What would that even mean? A forked Java's strings could insist their implementation is UTF-8 encoded bytes, but that's strings, you're talking about char. Do you want char just to be a byte, like in C ? But Java already has a byte type.
- LukeShu 3y agoIn my forked Java, a char would be UTF-32, like in Go. (And String would be UTF-8 encoded bytes, as you say.)
- kiitos 3y agoGo has a concept of a byte, and a concept of a rune, but no concept of a char.
- ElectricalUnion 3y agoA 32 bit "wide char" is both very wasteful under normal use and mostly useless for grapheme cluster (what business people probably actually mean by "character") handling.
- kgeist 3y agoIn Go, strings are UTF8, but when you iterate over UTF8 characters like this: for _, c := range str { } variable "c" is 32-bit. So it's not really wasteful, it fits in a register. It can be wasteful if you declare a slice of characters, like []rune, but I don't remember ever seeing it in practice.
- aardvark179 3y agoIn Java strings are stored in the heap either as compact strings for cases where everything fits in 8 bits, or as UTF-16. Iterating over them using chars will give you those 16 bit things, but there are other methods you can use to iterate over them as code points, just like you can in Go. The old char behaviour may not be what you really want, but it can’t be changed without breaking existing code.
- LukeShu 3y agoRight. But this thread is about "OK, but what if we forked Java and could make breaking changes like that." Plus, I'd bet that most (but of course, not all) code that iterates over a Java String using UTF-16 chars is broken; failing to properly handle surrogate pairs, and should instead use those iterate-over-them-as-codepoints/UTF-32 methods.
- erik_seaberg 3y agoMost of the things you could do with a single codepoint aren’t valid unless you search for a group of unicode.IsMark(c) and process them together with the previous codepoint. Now I’m concerned that I don’t see that in the standard library. https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...
- ElectricalUnion 3y agoIt is not worth it because: * If you're not really serious about using strings, why would you care? * If you're really serious about actually doing things with Strings, you are gonna need to reimplement International Components for Unicode (or something similar in scope), and you have a free, libre and already working one for the "UTF-16 String Java" already. You don't have a working one for your custom fork with custom String handling.
- kevin_thibedeau 3y agoAlready done: J++ begat C#.
- Pet_Ant 3y agoIf I had my druthers, String would be an interface with UTF-8, UTF-16, and UTF-32 (and UTF-7 on April Fool's Day) implementation classes. Then add Byte1, Byte2, and Byte4 as _unsigned_ primitives. Maybe have a wrapper class CodePoint to allow abstracting over all of them.
- josho 3y agoI agree but wonder if JVM optimizations could do away with the need to complicate the programmer experience.
- colejohnson66 3y agoJava can't even bother adding unsigned integers or integer types smaller than 32-bits. How long have we been waiting for non-boxed primitives and runtime generics (Valhalla)? C# has had all (except generics) since day one. New "char" types aren't happening anywhere in the near future, unfortunately.
- marginalia_nu 3y ago> integer types smaller than 32-bits ... other than short and byte, what are you missing? nibble? https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html https://docs.oracle.com/javase/tutorial/java/nutsandbolts/da...
- Delk 3y ago> or integer types smaller than 32-bits Java's got byte (8 bits) and short (16 bits)? (Still signed, though, of course.) Unless those are actually padded to 32 bits in the VM or something.
- vips7L 3y agoDelivered almost 3 releases ago.
- cesarb 3y agoIf you follow the LTS releases, it will be delivered only on the next release, expected around two months from now.
- bawolff 3y agoNow do javascript ;) [I mean,make the strings use utf-8 instead of utf-16, which is a bit different than TFA]
- schemescape 3y agoAm I reading correctly that characters are 16 bits? If so, is this just for the boundaries (e.g. file system, web)?
- TacticalCoder 3y agoJava predates Unicode 3.1 (I think I got the version correct: but basically when Java was created, Unicode had less than 65536 codepoints). So Java had 16 bits chars from the get go and to this day is still backward compatible with earlier Java code. You can also ask to get the "codepoint", which returns you an int. But, yup, the Java char primitive is 16 bits. It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.
- Quekid5 3y ago> But to be honest Unicode in itself is probably the biggest SNAFU of our entire field. Explain. EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!
- EnergyAmy 3y agoIt was always going to be messy to create something that covered all of the writing systems in the world, but things like this didn't help, and is probably what they're referring to: https://en.wikipedia.org/wiki/Han_unification https://en.wikipedia.org/wiki/Han_unification
- tialaramex 3y agoI can't explain what the previous poster was thinking, but Unicode is frustrating to work with. The problem is, that frustration just replicates the reality of the human writing systems Unicode captures. Nobody invented weird nonsense to make your life harder in Unicode, millions of humans developed a variety of weird nonsense to make their lives easier, over thousands of years and Unicode captures most of it. Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.
- smarks 3y agoThe model of a Java String, as presented via the API, is as an array of 16-bit unsigned integers. Strings use a UTF-16-like encoding to represent non-BMP Unicode characters. The main difference from UTF-16 is that illegal sequences of 16-bit values (such as unpaired surrogates) are permitted. The "UTF-8 by Default" refers to the default character set used for decoding and encoding, when Java Strings are read and written. Prior to JEP 400, the default character set was "platform specific." It was often UTF-8, but sometimes it was not, in which case hijinks ensued. As noted elsewhere the internal representation of Java Strings is either UTF-16-like or is Latin-1, if all the characters of the String can be encoded in Latin-1. We think about using UTF-8 as an internal representation constantly. Doing so would potentially save space and reduce decoding/encoding overhead. Unfortunately doing this is difficult. The key issue is that tons of code out there treats a String as a UTF-16 array (since that's what's in the API) so there are coding patterns like this: for (int i = start; i < end; i++) { char ch = str.charAt(i); // do something with ch } If the internal representation were UTF-8, simplistically, `charAt(i)` would change from O(1) to O(N). Of course there are cleverer things one could do, such as converting to UTF-16 lazily. Now `charAt()` allocates memory and has variable latency. Well then partial conversion could be done and the current iteration point could be cached. This might work, but now String has state, and it's thread-specific as well. Etc.
- Phrodo_00 3y ago> If the internal representation were UTF-8, simplistically, `charAt(i)` would change from O(1) to O(N). 16 bits is not enough space to store all unicode characters. Does that mean that java's charAt won't join chars for codepoints over U+FFFF? Edit: Yeah, that's exactly what happens[1]. That's not a very nice implementation at all . > Because 16-bit encoding supports 2^16 (65,536) characters, which is insufficient to define all characters in use throughout the world, the Unicode standard was extended to 0x10FFFF, which supports over one million characters. The definition of a character in the Java programming language could not be changed from 16 bits to 32 bits without causing millions of Java applications to no longer run properly. To correct the definition, a scheme was developed to handle characters that could not be encoded in 16 bits. > The characters with values that are outside of the 16-bit range, and within the range from 0x10000 to 0x10FFFF, are called supplementary characters and are defined as a pair of char values. [1] https://docs.oracle.com/javase/tutorial/i18n/text/unicode.html https://docs.oracle.com/javase/tutorial/i18n/text/unicode.ht...
- daniel_grady 3y agoIt’s exciting. Between this and lambdas, Java will soon be as good as Python was ten years ago.
- kaba0 3y agoAt least Java has proper syntax for lambdas. But language wars are almost always meaningless, both have their places and they definitely don’t occupy the same niche.