4 ms·

The fundamental question is: what is the filesystem for? If it's for users to store documents, you want something like BeFS, with slower performance but more f

by panic 9y ago

The fundamental question is: what is the filesystem for? If it's for users to store documents, you want something like BeFS, with slower performance but more features. If it's for programs to persist data, you want something minimal which you can build database libraries on top of. Trying to do both at the time pulls you toward a mediocre middle ground. Operating systems should really have two separate systems, one for user documents and one for low-level persistence (the former probably being built on top of the latter). Programs needing plain persistent storage shouldn't have to pay for nice features at the document level, and we shouldn't have to compromise the user experience of document browsing to get high-performance persistence.

kobeya 9y ago
They do. That's what a modern file system is. The on disk data structures and related code is usually less complex than the higher level software features. It's just that unfortunately these layers aren't often exposed for use (ZFS is the exception). If I were writing a file system from scratch, and I have thought about this, I would make a base layer that did nothing more than provide a hardware independent interface to block devices plus B-tree and PATRICIA index implementation that includes block-level features such as RAID, snapshots, log structured updates, etc. Then on top of that you can either write a fast DBMS for a BeOS like experience, or a POSIX compatibility layer that makes a more traditional hierarchical interface.
kalleboo 9y ago
You can look at what Apple does for an example of how you compromise on a higher level than the filesystem. They offer regular file storage (in the file system), a structured object storage API (Core Data, which is really just built ontop of SQLite databases stored in the file system), and an cross-application metadata and search index (Spotlight). Software that wants to store structured documents without thinking up their own file format can use Core Data, and still get their documents indexed properly. Software that's cross-platform or has to support existing file formats can use regular documents and supply a file metadata/content indexer to Spotlight.
cturner 9y ago
I have been thinking about another tradeoff recently. For lots of rapid access, you want your filesystem to behave like a character device, and to be able to async dispatch all operations (including open and close). For fewer accesses to large data selections you want block devices that map to memory (in the process, you lose async options).
eru 9y ago
> Operating systems should really have two separate systems, [...] That was the promise of exo-kernels: remove all abstractions from the kernel, leaving only what's required for secure multiplexing of resources, and then let user level libraries handle abstractions and cross-platform concerns. That includes multiple different file system in user land.

by panic 9y ago

kobeya 9y ago

They do. That's what a modern file system is. The on disk data structures and related code is usually less complex than the higher level software features. It's just that unfortunately these layers aren't often exposed for use (ZFS is the exception). If I were writing a file system from scratch, and I have thought about this, I would make a base layer that did nothing more than provide a hardware independent interface to block devices plus B-tree and PATRICIA index implementation that includes block-level features such as RAID, snapshots, log structured updates, etc. Then on top of that you can either write a fast DBMS for a BeOS like experience, or a POSIX compatibility layer that makes a more traditional hierarchical interface.

kalleboo 9y ago

You can look at what Apple does for an example of how you compromise on a higher level than the filesystem. They offer regular file storage (in the file system), a structured object storage API (Core Data, which is really just built ontop of SQLite databases stored in the file system), and an cross-application metadata and search index (Spotlight). Software that wants to store structured documents without thinking up their own file format can use Core Data, and still get their documents indexed properly. Software that's cross-platform or has to support existing file formats can use regular documents and supply a file metadata/content indexer to Spotlight.

cturner 9y ago

I have been thinking about another tradeoff recently. For lots of rapid access, you want your filesystem to behave like a character device, and to be able to async dispatch all operations (including open and close). For fewer accesses to large data selections you want block devices that map to memory (in the process, you lose async options).

eru 9y ago

> Operating systems should really have two separate systems, [...] That was the promise of exo-kernels: remove all abstractions from the kernel, leaving only what's required for secure multiplexing of resources, and then let user level libraries handle abstractions and cross-platform concerns. That includes multiple different file system in user land.