3 ms·
> UTF-8 is a variable length Unicode encoding like UTF-16, but defaults to a small one-byte code unit and has a famously elegant algorithm ... that is 100% com
by derleth 14y ago
> UTF-8 is a variable length Unicode encoding like UTF-16, but defaults to a small one-byte code unit and has a famously elegant algorithm
... that is 100% compatible with ASCII, and, therefore, is the only Unicode encoding scheme you can safely use in filenames and to send text through applications that don't know about Unicode at all.
The primary reason is that in UTF-32 and UTF-16, the byte 0x00 may appear in the encoding of characters other than '\0'. This is not something pre-Unicode applications can deal with, because in ASCII the byte 0x00 always means '\0', which was always used for string termination.
Therefore, any application that processes ASCII and leaves non-ASCII alone (common in the real world) is instantly compatible with UTF-8.
Here's a fascinating discussion about character encodings in filenames, including comments from Linus Torvalds and Theodore Ts'o:
http://yarchive.net/comp/linux/utf8.html http://yarchive.net/comp/linux/utf8.html
The upshot is, Linux, like most if not all Unix-like OSes, speaks bytestreams, and doesn't try to interpret characters. The only rules you need, therefore, are that filenames can't contain the byte 0x2f ('/' in ASCII) except as a path separator and can't contain the byte 0x00 at all. Therefore, you need a character encoding that will not use the bytes 0x2f or 0x00 to represent characters other than '/' and '\0', respectively. The only Unicode encoding that meets that constraint is UTF-8.
Also: The elegant algorithm means that it's statistically highly, highly improbable that any text that can be interpreted as valid UTF-8 isn't UTF-8. Given that it's entirely valid to treat ASCII as UTF-8, defaulting to UTF-8 is, as you say, a good thing to do.