5 ms·
I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16
by konstmonst 9y ago
I also think encoding doesn't belong into a file system. Let the names be arrays of bytes and leave the encoding to the people that use it, be it utf-8, utf-16 or something entirely different.
- cryptonector 9y agoZFS has an option to reject non-UTF-8. But if you disable that then it allows it, and for all UTF-8 it is normalization-preserving/insensitive. This is cheap and ideal. EDIT: i.e., ZFS doesn't know or care about encodings. Its n-p/n-i behavior means that for non-UTF-8 that happens to appear as valid UTF-8 there is some potential aliasing behavior going on, but this is exceedingly unlikely, and we decided that it was worth taking that risk.
- cryptonector 9y agoWhat really doesn't belong is keeping track of what codeset any given name/string uses. It's ETOOHARD.
- rtpg 9y agothe problem that arrives there almost instantly is that people want to see a list of filenames. If you're ever going to do some sort of "displaying" of data, you cannot store it as bytes. You need to know what characters things are supposed to be presented as. You could imagine not settling on a specific encoding, but you must know the encoding. Unless your plan is to show a list of numbers to users.
- cryptonector 9y agoRight. And it's just rather difficult for the kernel to know the user-land LC_CTYPE setting, so it doesn't, so it doesn't know the encoding of any string, so the best thing to do is assume UTF-8. If you want some other encoding, then libc's syscall stubs will have to do codeset conversions, or if not then you have to make sure that you only use that codeset everywhere.
- thaumasiotes 9y agoThe filesystem can't be responsible for displaying anything, though. Displaying filenames is the job of the shell / window manager / etc. The filesystem is much better off handling filenames as a number of arbitrary bytes. Let people who want to put weird bytes in their filenames see ugly filenames along the lines of "\x00 Can you see this?"
- rtpg 9y agoOK so I move a file from my hard drive to a USB drive, and give it to a friend. Can they not read the filenames anymore because their shell/window manager is different?
- cryptonector 9y agoRight! Or worse, you speak multiple languages, or learn new ones, and need to... switch codesets? How do you then access your old files?! We must switch to Unicode. Full stop. If there are imperfections in Unicode script support, then we must fix those, but otherwise we must adopt Unicode.
- cryptonector 9y agoSoftware will need to know what codeset and encoding those arbitrary bytes are in!