8 ms·
Make JDK source code UTF-8
- java-man 4y agoExcellent idea!
- PaulHoule 4y agoI had no idea it wasn't. I write Java source with UTF-8 characters whenever I can.
- remexre 4y agoThe source code of the JDK itself, not the input it accepts.
- PaulHoule 4y agoA lot of that source is Java and thus accepted by javac, though looking at it https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javac.html https://docs.oracle.com/javase/7/docs/technotes/tools/window... -encoding encoding Set the source file encoding name, such as EUC-JP and UTF-8. If -encoding is not specified, the platform default converter is used. where that last bit about the platform default converter is something I'd expect from a language that comes out of a squishier (like Python or POSIX) culture.
- vbezhenar 4y agoPlatform-default encoding was Java scourge for many years. It's finally changing I think and default encoding will be UTF-8 soon.
- saghm 4y agoIf anything, I would have assumed UTF-16, since I thought that was the standard way Java dealt with strings
- smarks 4y agoInternally, the model of a String is UTF-16-like, a sequence of unsigned 16-bit `char` values. However, it's not strictly UTF-16 because a Java String permits unpaired surrogates and invalid code points. Externally, Java has historically dealt with whatever character set was in use on whatever platform it was running on. As time as gone on, most platforms have converged on UTF-8 for external data storage and interchange. Time for Java to do that too, at least by default.
- jillesvangurp 4y agoThe common mistake is relying on any 'default' for this. There's no such thing. This has caused me a huge amount of headaches over the years in the form of messed up strings in data exports, databases, or API outputs. There's always one place where somebody forgets to specify Utf-8 and relies on the default. Which is often not Utf-8 on lots of operating systems depending on how they are configured. Even Linux has issues with this. Even in 2023. Most distributions set a default encoding of UTF-8 of course but if you start your server early in the boot process as root, none of that may be applied. If you write your own systemd scripts, be sure to make this explicit as well. I've found this out the hard way where the default was latin-1 and combined with somebody not doing what they should be doing, you end up dealing with corrupted data. It's why not specifying a encoding when decoding bytes to a string is a mistake in Java that things like spotbugs check for and warn against. It's one of those things that IMHO should have been deprecated in Java ages ago. Instead they keep on adding more APIs that do the same thing. Removing it would probably break lots of code. But the point is that that code should already be considered broken and should be fixed. Thankfully, Kotlin uses sensible default parameters in Kotlin extension functions that do similar things for this (of Utf-8). If you want something else, you have to be explicit about it. That's the only sane way to deal with character encodings IMHO. As long as you don't use Java APIs directly, things are fine.
- PaulHoule 4y agoI think the industry is moving towards UTF-8 everywhere as a default however the platform is misconfigured. Python certainly is. I think it is rare but not unheard of to find files in a different encoding and if that is the case it is not deliberate but an accident because of a misconfigured default encoding.
- xyst 4y agoside note: 8M+ issues just in the openjdk project so much history here, or just junk.
- deleted 4y ago[deleted]
- smarks 4y agoLots of useful history. The bug in question has a number in the 8,000,000 range, but that doesn't mean there are 8M bugs. The bug numbering was carried over from the old Sun-internal bug database. That bug database was shared across a large number of Sun projects, not just Java, so it went through the numbers pretty quickly. I'm not entirely sure, but I think the numbering might have started at 1,000,000 in order to make sure the bug IDs all had seven digits. In addition, there were several migrations to different databases, where "new" bugs were created at a particular range (say, 4,000,000) to distinguish them from "old" bugs that had been migrated from the previous system. Thus, there are large gaps in the numbering. When OpenJDK migrated from the old Sun bug database to JIRA, something similar was done. New bugs were created starting in the 8,000,000 range. Most of the old Java-related bugs were migrated from the Sun bug database into JIRA, preserving their existing numbers. There are bugs in the 1,000,000 range that date back to 1994, predating Java 1.0, and well before Java was open-sourced. The lowest-numbered bug in the current database is this one from 1994: https://bugs.openjdk.org/browse/JDK-1169723 https://bugs.openjdk.org/browse/JDK-1169723 (This includes only JDK bugs. There are other projects in the same JIRA instance that have different numbering schemes.) Finally, a single bug can generate multiple items in the JIRA database. For example, if a bugfix needs to be backported to an older release, this generates subtasks for each targeted release. There are about six actively maintained releases, so there's a bit of a multiplier effect. (Not every bugfix is backported, though.) There are currently about 318,000 bugs (open and closed) in the JDK project.
- WirelessGigabit 4y agoWarning, controversial: I'm actually against this. While I believe we should have a standard, I feel that having UTF-8 in source code brings more risks than benefits. [0] Next to those additional risks it also limits discoverability if `fiancee` now is written as `fiancée` (yea I had to c/p that from somewhere). Searching for the former does not discover the latter. Lastly, there is the issue of Intellisense, or whatever it is called in many languages. I have seen codebases in English with the aforementioned `é` in a function name. The only way for me to select that function name was with the arrow keys / mouse. I couldn't type it on my QWERTY. And yes, I know that there are codebases which are non-English, and not even Latin. Those are very valid concerns to which I admittedly have no answer to. [0] https://krebsonsecurity.com/2021/11/trojan-source-bug-threatens-the-security-of-all-code/ https://krebsonsecurity.com/2021/11/trojan-source-bug-threat...
- Raymonf 4y agoé is a valid character in latin-1, the current encoding in "many cases".
- vips7L 4y agoThis isn’t about your codebase. It’s about the JDK’s codebase.
- francoisdevlin 4y agoI once wrote a math library with using the letters delta and epsilon. It was cute, but I had to copy/paste every time. Never, ever again.
- EntrePrescott 4y agodepending on your OS / windowing system, you actually have easy ways to quickly get greek letters out of your (non-greek) keyboard without having to copy-paste them: my favorite way is to assign a key or key combination of your choice to act as so-called dead_greek modifier, and then just press that modifier before a latin letter key to get the corresponding greek letter. For example, under linux/xorg, if you wanted AltGr+g to be your dead_greek, you can use xmodmap to set it so: xmodmap -e "$(xmodmap -pke | grep "keycode\s\+42" | sed -E 's/=((\s\S){4})\s\S+/=\1 dead_greek/')" you'll find more on that here: https://askubuntu.com/questions/787113/compose-dead-greek-with-compose-key https://askubuntu.com/questions/787113/compose-dead-greek-wi... alternative 1: assign a key as a direct modifier (not a dead key) and add key combination defs for that to generate the characters you want … alternative 2: switchable keyboard layouts
- xvilka 4y agoI wish Windows had better UTF-8 (and Unicode overall) support. The system-wide UTF-8 locale is still in beta after so many decades of suffering. Windows Console/Terminal/PowerShell still have constant issues with CJK.
- mouse_ 4y agoMsys2 is/has always been cool tho
- samsaga2 4y agoOur software is utf8 and we have problems open files with accents characters. We have to convert the path strings before open a file.
- ulrikrasmussen 4y agoWindows is still using WTF-16 for filenames, right? https://simonsapin.github.io/wtf-8/#wtf-16 https://simonsapin.github.io/wtf-8/#wtf-16
- gwbas1c 4y agoI recently did something similar in C#. I ended up using .editorconfig to force an error in CI (and in Visual Studio) when a file isn't utf8; and to force Visual Studio to create new files in utf8.
- benatkin 4y agoFor a minute I was wondering why they weren't making it UTF-7 like everything else. Then I realized that it's UTF-8. I'm sleepy haha. However, in UTF-8, 7 bits are what is available to distinguish between characters that take only one byte. U+0000 through U+007F, or 0x00 through 0x7F. Ones starting with the first byte 0x80 or higher are chonkier. https://www.youtube.com/watch?v=NQFMeyWVe3g https://www.youtube.com/watch?v=NQFMeyWVe3g
- bigDinosaur 4y ago7 bits to encode characters of one byte is for ASCII compatibility, ASCII of course being a 7 bit encoding.
- benatkin 4y agoI'm not sure that's the only reason. That is also lucky :) I guess I will have to read about those who made it. (The official story may not tell all.) It's not ideal just for that reason but because 7 bits is a good amount to preserve for other characters, as well as the simplicity of only having to check one bit to tell it to look at the next byte to find what character it is. It could have been all but one to say that it isn't a single byte character. Then there would be 255 single byte characters and 256 two byte characters, minus however many you use to specify that there are more bytes. It would be easy to make an encoding format where 0x00 or 0xFF (0b00000000 or 0b11111111) say to look at the next byte, but that would suck as a general text encoding format.
- pabs3 4y agoTheir checklist is missing one item; look for mojibake and figure out how to re-encode it in UTF-8. I had to do this recently, and wow it was hard to figure out the right characters! I think I ended up Googling for the string and using that.
- mook 4y agoI've had good luck years ago just duplicating the string a few times and throwing it at Mozilla's character set detection stuff (there's a "chardet" python library implementing that, IIRC). My context was song metadata (title, artist, etc.) so your mileage might vary.
- fithisux 4y agoActually I like the idea.
- BiteCode_dev 4y agoIt's been the case for python for a while and nothing bad happened. Nobody started to use accentuated variables. But I finally could put my name in comments without having to put a # coding header, which is nice.