6 ms·

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sa

Filesystems seem so fiddly and broken.. is it very difficult to make a small layer over filesystems that provides sane semantics? Like one that handles paths sanely, handles fclose()/fsync() properly, lets you control when things are buffered/flushed etc? Even a broken API with clear, modern documentation enumerating all the fiddly cases would be a huge step forward from digging through random mailing lists on sites with UX from the 90s. Has anyone tried this? Is it possible with FUSE? I would love to hear from people who know about this stuff - what are the obstacles? Or do you think FSs are fine the way they are?

chungy 9y ago
ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)
- cryptonector 9y ago
  It does more than that! It also does form-insensitive directory lookups.
  - chungy 9y ago
    Only if you set the normalization parameter. Otherwise it'll guarantee only that a file name is UTF8-strict and not that there aren't two files named "é" with different normalization. Honestly, normalization should probably always be set. It gets way more confusing than case-sensitivity (which can also be changed on ZFS!) already is.
ManuelKiessling 9y ago
sqlite?
cryptonector 9y ago
ZFS handles internationalization about as well as could be hoped for. You can forbid non-UTF-8 strings (note that ZFS doesn't know if some string that is valid UTF-8 is actually encoded in UTF-8 -- it might not be), and ZFS does form-insensitive directory lookups, so if you copy some normalized-to-NFD files from OS X, it will work out fine. Making filesystems codeset-aware is not worth the trouble. It's best instead to just use UTF-8 locales everywhere. If you need to deal with other codesets, convert as needed, but don't use non-UTF-8 locales. On ZFS with a fast ZIL fsync()/sync() function a lot like write barriers, which is what we really need. Actually, what we really need is for all filesystem operations to be available with async system calls, write barriers included.
mehrdadn 9y ago
As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... except, of course, when they change. Now try actually formalizing what exactly that means from a global (multi-program) perspective!
- mook 9y ago
  Isn't that why openat() exists? Of course, that isn't used nearly as much because it's annoying to have to do things that way, but it seems like the sort of thing if you need it. The "open things by paths" thing, IIRC, is part of the reason Windows doesn't like to let you delete open files by default.
  - mehrdadn 9y ago
    Yeah, and Windows has NtOpenFile(OBJECT_ATTRIBUTES*) to let you specify a parent directory, but these are only at the syscall level, not at the standard application API level (Win32 or C APIs don't allow it). The entire model exposed to normal applications has this problem.
    - lostmsu 9y ago
      I though Windows won't let you move or rename a directory, if it has open files.
      - mehrdadn 9y ago
        Almost, but not quite. I just tried it to confirm: Windows lets you delete parent directories if they're reparse points. (I don't believe the deletion prevention was attempting to solve this problem in the first place, so this isn't really a bug or anything.)
Hermel 9y ago
I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.
- brazzy 9y ago
  At least since Java 7, the API has a concept of a filesystem which can create filesystem-specific Path objects and, if it encounters illegal characters, throws an InvalidPathException that tells you which character was illegal.
  - CodesInChaos 9y ago
    Sounds tricky to get right, considering the filesystem can change in the middle of a path.
    - brazzy 9y ago
      I haven't worked extensively with the API, but I believe that it represents filesystem instances accessible to the JVM, not abstract technical filesystems. So in the case of a Unix filesystem tree with various mount points, the FileSystem object would throw the exception iff the path you're trying to construct is illegal for the actual filesystem configuration in the system.
- klodolph 9y ago
  ":" is or is not a legal character in filenames on macOS depending on which API you use! Try saving a file or renaming a file to contain ":" using the GUI. It will not work. However, "/" is fine. Try saving a file or renaming a file to contain "/" using the CLI. It will not work. However, ":" is fine. A ":" in the CLI is translated to "/" in the GUI and vice versa. ":" was the directory separator in Mac OS 9 and earlier, which explains the behavior. The actual on-disk filenames will contain "/", which is translated to ":" for the POSIX API. This is for HFS and HFS+, not sure how APFS changes things.
ktpsns 9y ago
The uncomfortable generic answer to this: https://xkcd.com/927/ https://xkcd.com/927/
- majewsky 9y ago
  It even lists "character encodings" in the title as an example.
wongarsu 9y ago
What is sane path handling? Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windows Explorer) can't open extended-length paths. So you would be creating files that only select software can even open, and which the user can't browse without third party software. (An easier to ignore fun fact is that NTSF and Windows also support case sensitive names if you set the right flags in the file APIs. But nobody uses that, so it's probably save to ignore that (until somebody mounts EXT3 partitions in windows...))
- majewsky 9y ago
  > So you would be creating files that only select software can even open, and which the user can't browse without third party software. iOS suggests that people are ok with this. /s
- ryl00 9y ago
  There's support for >260 char path lengths now without the \\?\ prefix, but I think you need to modify the registry, plus the application needs to opt into the new behavior via a setting in its manifest.
- tinus_hn 9y ago
  You can use subst to open these paths. The better option is to not create them in the first place though.
jstimpfle 9y ago
The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore. Another way to put this is "shit in, shit out". So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least. When someone else modifies a directory tree at the same time, we're bound to run into problems. That's just how it is. Actually there are synchronization facilities (e.g. flock()) for some of these types of problems. But these are seldom used because they lead to other problems. I think they knew all that in the 70s, and just chose to be pragmatic about it.
- fiedzia 9y ago
  > So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. The problem with invalid filenames is that doing anything at all with them might be impossible (without escaping of fixing). Your output (gui or terminal) most likely uses utf8, you can't pass invalid filename to it, so you can't even display it. Also in most situation you can't just ignore something. Imagine a text editor where user tries to open invalid file, how do you "ignore" that? Many (if not most) application that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem.
  - jstimpfle 9y ago
    > Many (if not most) applications that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem. Yep, that's what I meant. You need to do what is appropriate to the situation. Ignoring / warning are only two possible handling strategies. In many situations, straight-out failing is another valid one. A file copy program should just not care and copy the darn thing. A text editor, basically the same, but maybe issue a warning. Different tasks have different requirements, like the filename being text, the filename not containing spaces, etc. Given that these requirements are somewhat arbitrary, I think it's a fine choice to just not put any non-technical constraints in the guts. There's an expression for it: Mechanism, not policy. You can always add policy on top, be it in the VFS layer, or as additional programs / classes of programs. So many potential bugs would be easily fixed if the shell glob ('*') was more configurable.
lmm 9y ago
What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared to deal with malformed names, because other programs will create files with malformed names.
- fiedzia 9y ago
  > you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics In many cases they offer the exact same semantics (damn you, mysql). Also they store data in files, and those files have to be named, so... back to square one.
  - lmm 9y ago
    > In many cases they offer the exact same semantics (damn you, mysql). I mean bad databases exist, sure, but the solution to that is to not use those. > Also they store data in files, and those files have to be named, so... back to square one. Not really - the database developers have handled the fsync, buffering and what-have-you for you, so you don't have to deal with them. (And FWIW serious databases generally offer the option of storing data on raw partitions).

chungy 9y ago

ZFS has a `utf8only` property that constrains filenames to be UTF-8 only... :)

cryptonector 9y ago
It does more than that! It also does form-insensitive directory lookups.
- chungy 9y ago
  Only if you set the normalization parameter. Otherwise it'll guarantee only that a file name is UTF8-strict and not that there aren't two files named "é" with different normalization. Honestly, normalization should probably always be set. It gets way more confusing than case-sensitivity (which can also be changed on ZFS!) already is.

ManuelKiessling 9y ago

sqlite?

cryptonector 9y ago

ZFS handles internationalization about as well as could be hoped for. You can forbid non-UTF-8 strings (note that ZFS doesn't know if some string that is valid UTF-8 is actually encoded in UTF-8 -- it might not be), and ZFS does form-insensitive directory lookups, so if you copy some normalized-to-NFD files from OS X, it will work out fine. Making filesystems codeset-aware is not worth the trouble. It's best instead to just use UTF-8 locales everywhere. If you need to deal with other codesets, convert as needed, but don't use non-UTF-8 locales. On ZFS with a fast ZIL fsync()/sync() function a lot like write barriers, which is what we really need. Actually, what we really need is for all filesystem operations to be available with async system calls, write barriers included.

mehrdadn 9y ago

As far as I can tell, file systems are inherently broken by design; this isn't an implementation issue. For example, the notion of finding a file by its path is just riddled with race conditions. If you create files /a/b/c and /a/b/d, are c and d necessarily in the same directory? Not really, because someone could have moved around the parent directories in between. But we conveniently assume paths stay the same... except, of course, when they change. Now try actually formalizing what exactly that means from a global (multi-program) perspective!

mook 9y ago
Isn't that why openat() exists? Of course, that isn't used nearly as much because it's annoying to have to do things that way, but it seems like the sort of thing if you need it. The "open things by paths" thing, IIRC, is part of the reason Windows doesn't like to let you delete open files by default.
- mehrdadn 9y ago
  Yeah, and Windows has NtOpenFile(OBJECT_ATTRIBUTES*) to let you specify a parent directory, but these are only at the syscall level, not at the standard application API level (Win32 or C APIs don't allow it). The entire model exposed to normal applications has this problem.
  - lostmsu 9y ago
    I though Windows won't let you move or rename a directory, if it has open files.
    - mehrdadn 9y ago
      Almost, but not quite. I just tried it to confirm: Windows lets you delete parent directories if they're reparse points. (I don't believe the deletion prevention was attempting to solve this problem in the first place, so this isn't really a bug or anything.)

Hermel 9y ago

I think Java does that. Thanks to its built-in libraries, you can write code that works on multiplie platforms with different file systems. However, you still need to know yourself which characters are legal in which system (for example, ‘:’ is not a legal character in file names on windows, but it is on OS X.

brazzy 9y ago
At least since Java 7, the API has a concept of a filesystem which can create filesystem-specific Path objects and, if it encounters illegal characters, throws an InvalidPathException that tells you which character was illegal.
- CodesInChaos 9y ago
  Sounds tricky to get right, considering the filesystem can change in the middle of a path.
  - brazzy 9y ago
    I haven't worked extensively with the API, but I believe that it represents filesystem instances accessible to the JVM, not abstract technical filesystems. So in the case of a Unix filesystem tree with various mount points, the FileSystem object would throw the exception iff the path you're trying to construct is illegal for the actual filesystem configuration in the system.
klodolph 9y ago
":" is or is not a legal character in filenames on macOS depending on which API you use! Try saving a file or renaming a file to contain ":" using the GUI. It will not work. However, "/" is fine. Try saving a file or renaming a file to contain "/" using the CLI. It will not work. However, ":" is fine. A ":" in the CLI is translated to "/" in the GUI and vice versa. ":" was the directory separator in Mac OS 9 and earlier, which explains the behavior. The actual on-disk filenames will contain "/", which is translated to ":" for the POSIX API. This is for HFS and HFS+, not sure how APFS changes things.

ktpsns 9y ago

The uncomfortable generic answer to this: https://xkcd.com/927/ https://xkcd.com/927/

majewsky 9y ago
It even lists "character encodings" in the title as an example.

wongarsu 9y ago

What is sane path handling? Alone on windows, the maximum path length is 260 characters, except when you use extended-length paths which have a 4 character prefix and a maximum length of 32,767 characters. A sane API for reading files probably converts your paths to extended-length paths. But if you do the same thing for writing files your users start calling you insane again, because most of Windows (including Windows Explorer) can't open extended-length paths. So you would be creating files that only select software can even open, and which the user can't browse without third party software. (An easier to ignore fun fact is that NTSF and Windows also support case sensitive names if you set the right flags in the file APIs. But nobody uses that, so it's probably save to ignore that (until somebody mounts EXT3 partitions in windows...))

majewsky 9y ago
> So you would be creating files that only select software can even open, and which the user can't browse without third party software. iOS suggests that people are ok with this. /s
ryl00 9y ago
There's support for >260 char path lengths now without the \\?\ prefix, but I think you need to modify the registry, plus the application needs to opt into the new behavior via a setting in its manifest.
tinus_hn 9y ago
You can use subst to open these paths. The better option is to not create them in the first place though.

jstimpfle 9y ago

The simple thing to go about this is to make assumptions that do not always hold technically. And to be ok when these assumptions break - in this case you simply can't deliver your promises anymore. Another way to put this is "shit in, shit out". So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. That's what I would do if I wrote a music tagger, at least. When someone else modifies a directory tree at the same time, we're bound to run into problems. That's just how it is. Actually there are synchronization facilities (e.g. flock()) for some of these types of problems. But these are seldom used because they lead to other problems. I think they knew all that in the 70s, and just chose to be pragmatic about it.

fiedzia 9y ago
> So: When receiving a filename that is not UTF-8, one could just emit a warning and ignore the file. The problem with invalid filenames is that doing anything at all with them might be impossible (without escaping of fixing). Your output (gui or terminal) most likely uses utf8, you can't pass invalid filename to it, so you can't even display it. Also in most situation you can't just ignore something. Imagine a text editor where user tries to open invalid file, how do you "ignore" that? Many (if not most) application that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem.
- jstimpfle 9y ago
  > Many (if not most) applications that process filenames do rely on those being valid text at least in some codepaths and they simply cannot work otherwise. The proper solution is to fail and let the user fix the problem. Yep, that's what I meant. You need to do what is appropriate to the situation. Ignoring / warning are only two possible handling strategies. In many situations, straight-out failing is another valid one. A file copy program should just not care and copy the darn thing. A text editor, basically the same, but maybe issue a warning. Different tasks have different requirements, like the filename being text, the filename not containing spaces, etc. Given that these requirements are somewhat arbitrary, I think it's a fine choice to just not put any non-technical constraints in the guts. There's an expression for it: Mechanism, not policy. You can always add policy on top, be it in the VFS layer, or as additional programs / classes of programs. So many potential bugs would be easily fixed if the shell glob ('*') was more configurable.

lmm 9y ago

What are you trying to achieve? If you want to store data for your own application, you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics. The only reason to interact with the OS-level filesystem is to use it to communicate with other programs, in which case the insanity is necessary: if you want to e.g. read files created by other programs, you have to be prepared to deal with malformed names, because other programs will create files with malformed names.

fiedzia 9y ago
> you'd use something higher-level than a filesystem (e.g. a database); those exist and offer sane semantics In many cases they offer the exact same semantics (damn you, mysql). Also they store data in files, and those files have to be named, so... back to square one.
- lmm 9y ago
  > In many cases they offer the exact same semantics (damn you, mysql). I mean bad databases exist, sure, but the solution to that is to not use those. > Also they store data in files, and those files have to be named, so... back to square one. Not really - the database developers have handled the fsync, buffering and what-have-you for you, so you don't have to deal with them. (And FWIW serious databases generally offer the option of storing data on raw partitions).