3 ms·

I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly a

I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly also ext4, which introduced a shift in paradigm, which broke old applications (or at least broke their performance, for example dpkg). - We're talking about a hierarchical file system, meaning it's easy for humans to find data, but terrible for machines because they must hunt pointers to get to files. Similar problem for syncing a bunch of logically connected files if they are not stored in a single directory. (How do you atomically commit multiple changes across directories?) What would be a better abstraction or non-abstraction (that's still a hierarchical file system)?

Ericson2314 11y ago
A friend of mine had the realization that all optimization boils down to making lower layers understand higher layers' concerns, or higher level layers understand on lower levels' concerns. Here are some examples off the top of my head: POSIX fails as a high level interface: - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) - Overall pushes people to maintain their own structure within files rather than use FS's trees. POSIX fails as a low level interface: - No way to hint or control caching along the memory hierarchy. - Block size, locality out of control. - Can't statically disallow various non-free actions which are costly (ability to expand file), can only hope that not using them does incur penalty. Perhaps ZFS an BTRFS made other approaches more viable/obvious, but these weakness are inherent to the API itself and the high level stuff was noticeably missing from the get-go.
- jstimpfle 11y ago
  > - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. There is nothing that prevents you from implementing these in userland. Does not belong in the kernel, since the kernel can't know what are your atoms that must be atomically committed. Research how databases do it. > - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) You can check what device a file belong to with stat(2). It's the st_dev member. You can also check it from the shell with the stat command. > - Overall pushes people to maintain their own structure within files rather than use FS's trees. And that's entirely ok. Hierarchies are not for databases. Database-y problems are not the problems that the POSIX fs solves. > - Block size, locality out of control. I actually heard that Unix filesystems have traditionally been quite good at preserving locality. That's why I don't know of any defrag tool for e.g. ext3. Overall, if the FS does not solve your problems, implement your own abstraction. That's perfectly ok.
theoh 11y ago
I think this is a bad option from a concurrency point of view, but why not one of the relational database alternatives for maintaining a hierarchy? http://stackoverflow.com/questions/4048151/what-are-the-options-for-storing-hierarchical-data-in-a-relational-database http://stackoverflow.com/questions/4048151/what-are-the-opti...
the_why_of_y 11y ago
It wasn't ext4 that introduced a paradigm shift but ext3: the fsync() implementation in ext3 by design wrote all buffered writes for the whole filesystem, not just a single file, hence it was so horribly slow that it taught a generation of application programmers never to call fsync() to avoid multi-second pauses.

Ericson2314 11y ago

A friend of mine had the realization that all optimization boils down to making lower layers understand higher layers' concerns, or higher level layers understand on lower levels' concerns. Here are some examples off the top of my head: POSIX fails as a high level interface: - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) - Overall pushes people to maintain their own structure within files rather than use FS's trees. POSIX fails as a low level interface: - No way to hint or control caching along the memory hierarchy. - Block size, locality out of control. - Can't statically disallow various non-free actions which are costly (ability to expand file), can only hope that not using them does incur penalty. Perhaps ZFS an BTRFS made other approaches more viable/obvious, but these weakness are inherent to the API itself and the high level stuff was noticeably missing from the get-go.

jstimpfle 11y ago
> - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. There is nothing that prevents you from implementing these in userland. Does not belong in the kernel, since the kernel can't know what are your atoms that must be atomically committed. Research how databases do it. > - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) You can check what device a file belong to with stat(2). It's the st_dev member. You can also check it from the shell with the stat command. > - Overall pushes people to maintain their own structure within files rather than use FS's trees. And that's entirely ok. Hierarchies are not for databases. Database-y problems are not the problems that the POSIX fs solves. > - Block size, locality out of control. I actually heard that Unix filesystems have traditionally been quite good at preserving locality. That's why I don't know of any defrag tool for e.g. ext3. Overall, if the FS does not solve your problems, implement your own abstraction. That's perfectly ok.

theoh 11y ago

I think this is a bad option from a concurrency point of view, but why not one of the relational database alternatives for maintaining a hierarchy? http://stackoverflow.com/questions/4048151/what-are-the-options-for-storing-hierarchical-data-in-a-relational-database http://stackoverflow.com/questions/4048151/what-are-the-opti...

the_why_of_y 11y ago

It wasn't ext4 that introduced a paradigm shift but ext3: the fsync() implementation in ext3 by design wrote all buffered writes for the whole filesystem, not just a single file, hence it was so horribly slow that it taught a generation of application programmers never to call fsync() to avoid multi-second pauses.