5 ms·
> Unix paths don’t need to be valid UTF-8 Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming lan
by marcosdumay 7y ago
> Unix paths don’t need to be valid UTF-8
Yet, your shell will treat them like UTF-8 just as well. As will the standard library of almost every programming language, as you noticed.
If you open one such file in most text editors, they will render whatever is in it as UTF-8. If you use text manipulating utilities, they will work with it as if it was encoded in UTF-8.
It's mostly the Linux kernel that disagrees. Everything else considers them UTF-8.
- arendtio 7y agoDoesn't it depend on your locales? At least for source-based Linux distributions (Gentoo, Exherbo) I remember that you have to define the locales you want to use and which ones should be the default. And when I build a system without UTF-8 locales, I doubt that the shell will treat paths as UTF-8.
- ordu 7y agoShell is like most of the programs doesn't need to bother about encoding of filenames. Mostly doesn't. I could use LANG=C and then TAB autocompletes filenames even cyrillic ones, because bash wouldn't mind encodings: terminal uses utf8, so it could output utf8 without any help from bash. Though it nevertheless pain to work with this sometimes, because readline fails to count visible characters (counting bytes instead). You type chars into command line, fill it to the end, then cursor jumps to the left side of the terminal and continues, placing characters over other characters. It is like \r used instead of \n. `LANG=C ls` tries to be smarter and uses escape-syntax for everything except printable ASCII characters. But other utilities from coreutils work even with a locale that doesn't match file name encoding. cp, mv, grep, ... The point is: it doesn't matter what encoding strings use until you tried to render string on a screen.
- Spivak 7y agoWhich is a silly position since the kernel is the only thing that matters. You're right that not too many people will complain if your program crashes on non-UTF-8 paths. Same with spaces in group names. 100% valid and accepted. Breaks a ridiculous amount of software if you actually do it. But that doesn't mean it's right. It just means that we have a calcified convention.
- marcosdumay 7y ago> narrow strings are considered UTF-8 by default almost everywhere It means that this is mostly true. I dunno what it should be. There are benefits and costs on both allowing and restricting the names. As well as there are good reason for the kernel alone to support them even tough all the userland doesn't. But it does mean that you just use UTF-8 and it's done.
- user5994461 7y agoWhat's right is that the software will be basically unusable outside of the US if it's crashing on non-ASCII characters in paths. ñ é ß characters appear in every text in Spanish, French, German, etc and they go in filenames too. Imagine if a table was called a tâble in English? Surely it would be outrageous to have software crash when you try to use the word tâble.
- rurban 7y agoExactly. And they still refuse to acknowledge that treating public names, like a file path, as binary only is a wellknown security issue. Names are identifiers and must be recognizable. With utf8 it is trivial to create similar looking names and fool the user to think it is a valid name. You know this concept from domain names, using punicode as escape mechanism. But both the kernel and the various libc's are too lazy to treat confusables with escapes, to normalize unicode or to use proper unicode security mechanisms for identifiers. Like mixing scripts, right to left and such. Eg searching a file path needs to follow unicode rules, as we are dealing with identifiers. I believe my libc, the safeclib, is the only one even offering such functionality. Likewise the presentation layer on the UI (shell, windows) doesn't present confusables as such, but happily takes i18n seriously. Convenience first, security last. Apple's previous HFS+ normalized names, the new one is insecure again.