4 ms·
The most hilarious issue with -Dfile.encoding is that if it set to, for example, UTF-8, then it is literally impossible to open files with names encoded in, say
by 0x0 11y ago
The most hilarious issue with -Dfile.encoding is that if it set to, for example, UTF-8, then it is literally impossible to open files with names encoded in, say, ISO-8859-1, which can happen on a Linux system where users use different LC_CTYPE.
You can do a "dir.listFiles()" and iterate it, and find that some of the entries are impossible to open because there is no way to represent the ISO-8859-1 bytes that make up the filename in a String object, and therefore no way to give the correct file name to the java io classes for opening.
- userbinator 11y agoYou can do a "dir.listFiles()" and iterate it, and find that some of the entries are impossible to open because there is no way to represent the ISO-8859-1 bytes that make up the filename in a String object, and therefore no way to give the correct file name to the java io classes for opening. That gives more weight to the argument that character encodings should be entirely a "presentation-layer" concern, and anything below that should treat strings as opaque byte sequences. There should also be a way to bypass that "presentation layer" transformation upon input.
- 0x0 11y agoAbsolutely, there's quite a few gaffes in java (other languages have similar issues too): * char being 16bit and thus can't hold a modern unicode character * lots and lots of string and byte stream methods that have variants without explicit charset parameters, making it very easy to accidentally have a hidden dependency on the implicit default charset (aka -Dfile.encoding) * Using strings instead of byte[] for filename values * Having to deal with UnsupportedEncodingExceptions as checked exceptions whenever referencing the UTF8 encoding
- jfim 11y agoThe reason for char being 16 bits is that the JLS predates Unicode 2.0 (1996), which added 32 bit codepoints. At that time, UCS-2 was widely believed to be sufficient to represent anything "international."
- tankenmate 11y agoIndeed, this is exactly what the Linux kernel does; it treats filenames like a binary blob, with the exception that the bytes 0x2f (ASCII '/') and 0x00 (ASCII NUL) are not accepted (regardless of where they are in the byte string).
- Kristine1975 11y ago> it treats filenames like a binary blob Which can result in files with seemingly identical names when one UTF-8-encoded file name uses combining characters and the other one uses precomposed ones. Not to mention the fact that e.g. Å is contained in Unicode as both "latin capital letter A with ring above" and "Angstrom sign". After you've solved encoding, next comes Unicode normalization ;-)
- aleh 11y agoIt is not a big deal as long as seemingly identical file names are treated by OS as different. Unicode normalization is not the only problem here, e.g. Latin 'a', 'e', 'T' are exactly the same as Cyrillic 'а', 'е', 'Т' in most fonts which makes it possible for two files to have seemingly same names even in some 8-bit encodings.
- Hondor 11y agoEven Latin 'I' and 'l' and the digit '1' are visually indistinguishable in some fonts! So are trailing spaces and different numbers of spaces. This is such a pervasive problem that maybe we can just give up and expect users to get used to it.
- Hondor 11y agoWow, so characters like U+022F (ȯ) and U+042F (Cyrillic letter Я) U+062F (Arabic letter د) are not allowed but nearly everything else is? Some of those are letters used in actual languages. That's sure to make people scratch their heads.
- pzb 11y ago
- masklinn 11y ago> which can happen on a Linux system where users use different LC_CTYPE. If only it were that simple. POSIX paths are encoding-less bags of bytes, you can literally find anything in them aside from NUL, LC_CTYPE isn't even a consideration.