4 ms·

The idea that filesystems are not just a flavor of database management systems was always a mistake. Maybe with micro-kernels we'll finally fix this.

7qW24A 1y ago
I’m a database guy, not an OS guy, so I agree, obviously… But what is the micro-kernel angle?
- packetlost 1y ago
  Likely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine. Persistent file systems are essentially key-value stores, usually with optimizations for enumerating keys under a namespace (also known as listing the files in a directory). IMO a big problem with POSIX filesystems is the lack of atomicity and lock guarantees when editing a file. This and a complete lack of consistent networked API are the key reasons few treat file systems as KV stores. It's a pity, really.
  - mrlongroots 1y ago
    > "Likely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine." "Userspace vs not" is a different argument from "consistency vs not" or "atomicity vs not" or "POSIX vs not". Someone still needs to solve that problem. Sure instead of SQLite over POSIX you could implement POSIX over SQLite over raw blocks. But you haven't gained anything meaningful. > Persistent file systems are essentially key-value stores I think this is reductive enough to be equivalent to "a key-value store is a thin wrapper over the block abstraction, as it already provides a key-value interface, which is just a thin layer over taking a magnet and pointing it at an offset". Persistent filesystems can be built over key-value stores. This is especially common in distributed filesystems. But they also circumvent a key-value abstraction entirely. > IMO a big problem with POSIX filesystems is the lack of atomicity Atomicity requires write-ahead logging + flushing a cache. I fail to see why this needs to be mandatory, when it can be effectively implemented at a higher layer. > This and a complete lack of consistent networked API A consistent networked API would require you to hit the metadata server for every operation. No caching. Your system would grind to a halt. Finally, nothing in the POSIX spec prohibits an atomic filesystem or consistency guarantees. It is just that no one wants to implement these things that way because it overprovisions for one property at the expense of others.
    - packetlost 1y ago
      > "Userspace vs not" is a different argument from "consistency vs not" or "atomicity vs not" or "POSIX vs not". Someone still needs to solve that problem. Sure instead of SQLite over POSIX you could implement POSIX over SQLite over raw blocks. But you haven't gained anything meaningful. This was an attempt to possibly explain the microkernel point GP made, which only really matters below the FS. > I think this is reductive enough to be equivalent to "a key-value store is a thin wrapper over the block abstraction, as it already provides a key-value interface, which is just a thin layer over taking a magnet and pointing it at an offset". I disagree with this premise. Key-value stores are an API, not an abstraction over block storage (though many are or can be configured to be so). File systems are essentially a superset of a KV API with a multitude of "backing stores". Saying KV stores are always backed by blocks is overly reductive, no? > Atomicity requires write-ahead logging + flushing a cache. I fail to see why this needs to be mandatory, when it can be effectively implemented at a higher layer. You're confusing durability for atomicity. You don't need a log to implement atomicity, you just need a way to lock one or more entities (whatever the unit of atomic updates are). A CoW filesystem in direct mode (zero page caching) would need neither but could still support atomic updates to file (names). > A consistent networked API would require you to hit the metadata server for every operation. No caching. Your system would grind to a halt. Sorry, I don't mean consistent in the ACID context, I mean consistent in the loosely defined API shape context. Think NFS or 9P. I also disagree with this to some degree: pipelined operations would certainly still be possible and performant but would be rather clunky. End-to-end latency for get->update-write, the common mode of operation, would be pretty awful. > Finally, nothing in the POSIX spec prohibits an atomic filesystem or consistency guarantees. It is just that no one wants to implement these things that way because it overprovisions for one property at the expense of others. I didn't say it did, but it doesn't require it which means it effectively doesn't exist as far as the users of FS APIs are concerned. Rename operations are the only API that atomicity is required by POSIX. However without a CAS-like operation you can't safely implement a lock without several extra syscalls.
      - mrlongroots 1y ago
        I was confused for a while about where this discussion was going, and what was the broader point. Will try to consolidate thoughts in the interest of making it clearer. You seem unhappy with POSIX because its guarantees feel incomplete and ad hoc (they are). You like databases because their guarantees are more robust (also true). DBMS over POSIX enables all the guarantees that you like. I'd want to invoke the end-to-end systems argument here and say that this is how systems are supposed to work: POSIX is closer to the hardware, and as a result it is messier. It's the same reason TCP in-order guarantees are layered above the IP layer. Some of your points re: how the lower layers work seem incorrect, but that doesn't matter in the interest of the big picture. The suggestion (re: microkernels) seems to be that POSIX has a privileged position in the system stack, and that somehow prevents a stronger alternative from existing. I'd say that your gripes with POSIX may be perfectly justified, but nothing prevents a DBMS from owning a complete block device, completely circumventing the filesystem. POSIX is the default, but it is not really privileged by any means.
- Ericson2314 1y ago
  The filesystem interface is only privilaged interface because it is the kernel knows about. E.g. you can already use FUSE and NFS to roll your own FS implementations, but those do not a microkernel make, because the OS is still in the way dictating the implementation. The safest way to put the FS on a level-playing field with other interfaces is to make the kernel not know about, just as it doesn't know about, say, SQL.
qwertox 1y ago
I can't agree with this. I like it that I can have all these tools which work with files and are tools which are not db-oriented, and the fact that there are different filesystems for different scenarios, that I can sandwich LVM between a FS and the block device. That /proc/ can pretend to be a FS because else we'd possibly end up with something like the Windows Registry for these operations, only managed through a database. Would you store all your ~/ in something like SQLite database?
- hdevalence 1y ago
  Yes, I would
- 90s_dev 1y ago
  > Would you store all your ~/ in something like SQLite database? Actually yeah that sounds pretty good. For Desktop/Finder/Explorer you'd just need a nice UI. Searching Documents/projects/etc would be the same just maybe faster? All the arbitrary stuff like ~/.npm/**/* would stop cluttering up my ls -la in ~ and could be stored in their own tables whose names I genuinely don't care about. (This was the dream of ~/Library, no?) [edit] Ooooh, I get it now. This doesn't solve namespacing or traversal.
  - fc417fc802 1y ago
    > This doesn't solve namespacing or traversal. That's "just" API. FS is "just" a KV store with a weird crufty API and a few extra tricks (bind mounts or whatever). I think the primary issue is the difference in performance between different strategies. It would be interesting to have a FS with different types of folders similar to how (for example) btrfs is generally CoW but you can turn that off via an attribute.
foobiekr 1y ago
Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.
- jonhohle 1y ago
  BeOS got it right with BeFS. An Email client was just a folder. MP3s could be sorted and filtered in the file system. https://news.ycombinator.com/item?id=12309686 https://news.ycombinator.com/item?id=12309686
  - int_19h 1y ago
    Windows does something similar with Explorer today when you open a folder that has mostly music files in it.
  - foobiekr 1y ago
    BeFS wasn't a database. It had indexed queries on EAs and they had the habit of asking application files to add their indexable content to the EAs. Internally it was just a mostly-not-transactional collection of btrees. There was no query language for updating files, or even inspecting anything about a file that was not published in the EAs (or implicitly do as with adapters), there were no multi-file transactions, no joins, nothing. Just rich metadata support in the FS.
    - Ericson2314 1y ago
      Yeah I am talking more deep architecture, and BeOS is more notable here mostly on just the user-interface level. However, I think it is reasonable to think that with way more time and money, these things would meet up. Think about it as digging a tunnel from both sides of the mountain.
      - foobiekr 1y ago
        Microsoft poured at least $100M into this hole with nothing to show for it.
01HNNWZ0MV43FF 1y ago
You could do a loopback network filesystem and make any user-space FS you want. That's what WSL does, and there's a Rust crate for it. Can't recall the name at all.
- Ericson2314 1y ago
  There is NFS and FUSE so you can write your own implementation, but you are still stuck with the interface that the kernel understands.
deleted 1y ago
[deleted]
runlaszlorun 1y ago
I've heard this mentioned a couple times but what would this look like functionality wise? A single "files" table with columns? Different tables for different categories of files? FTS? Something else?
- Ericson2314 1y ago
  See the other comments. The point is not a specific new interface, but a separation of concerns, and leveling the playing field. I'll try to do an example. The kernel doesn't currently know about SQL. Instead, you e.g. connect to a socket, and start talking to postgres. Imagine if FS stuff was the same thing: you connect to a socket, and then issue various command to read and write files. Ignore perf for a moment, it works right? Now, one counter-argument might be "hold up, what is this socket you need to connect to, isn't that part of a file system? Is there now an all-userspace inner filesystem, still kernel-supported 'meta filesystem'?" Well, the answer to that is maybe the Unix idea of making communication channels like pipes and (to a lesser extent) sockets, was a bad idea. Or rather, there may be nothing wrong with saying a directory can have a child which may be such a communication channel, but there is a problem with saying that every such communication channel should live inside some directory.
mrlongroots 1y ago
Thoughts: 1. Distributed filesystems do often use databases for metadata (FoundationDB for 3FS being a recent example) 2. Using a B+ tree for metadata is not much different from having a sorted index 3. Filesystems are a common enough usecase that skipping the abstraction complexity to co-optimize the stack is warranted

7qW24A 1y ago

I’m a database guy, not an OS guy, so I agree, obviously… But what is the micro-kernel angle?

packetlost 1y ago
Likely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine. Persistent file systems are essentially key-value stores, usually with optimizations for enumerating keys under a namespace (also known as listing the files in a directory). IMO a big problem with POSIX filesystems is the lack of atomicity and lock guarantees when editing a file. This and a complete lack of consistent networked API are the key reasons few treat file systems as KV stores. It's a pity, really.
- mrlongroots 1y ago
  > "Likely the idea that filesystems should run as userspace / unprivileged (or at least limited privilege) processes which would make them, ultimately, indistinguishable from a form of database engine." "Userspace vs not" is a different argument from "consistency vs not" or "atomicity vs not" or "POSIX vs not". Someone still needs to solve that problem. Sure instead of SQLite over POSIX you could implement POSIX over SQLite over raw blocks. But you haven't gained anything meaningful. > Persistent file systems are essentially key-value stores I think this is reductive enough to be equivalent to "a key-value store is a thin wrapper over the block abstraction, as it already provides a key-value interface, which is just a thin layer over taking a magnet and pointing it at an offset". Persistent filesystems can be built over key-value stores. This is especially common in distributed filesystems. But they also circumvent a key-value abstraction entirely. > IMO a big problem with POSIX filesystems is the lack of atomicity Atomicity requires write-ahead logging + flushing a cache. I fail to see why this needs to be mandatory, when it can be effectively implemented at a higher layer. > This and a complete lack of consistent networked API A consistent networked API would require you to hit the metadata server for every operation. No caching. Your system would grind to a halt. Finally, nothing in the POSIX spec prohibits an atomic filesystem or consistency guarantees. It is just that no one wants to implement these things that way because it overprovisions for one property at the expense of others.
  - packetlost 1y ago
    > "Userspace vs not" is a different argument from "consistency vs not" or "atomicity vs not" or "POSIX vs not". Someone still needs to solve that problem. Sure instead of SQLite over POSIX you could implement POSIX over SQLite over raw blocks. But you haven't gained anything meaningful. This was an attempt to possibly explain the microkernel point GP made, which only really matters below the FS. > I think this is reductive enough to be equivalent to "a key-value store is a thin wrapper over the block abstraction, as it already provides a key-value interface, which is just a thin layer over taking a magnet and pointing it at an offset". I disagree with this premise. Key-value stores are an API, not an abstraction over block storage (though many are or can be configured to be so). File systems are essentially a superset of a KV API with a multitude of "backing stores". Saying KV stores are always backed by blocks is overly reductive, no? > Atomicity requires write-ahead logging + flushing a cache. I fail to see why this needs to be mandatory, when it can be effectively implemented at a higher layer. You're confusing durability for atomicity. You don't need a log to implement atomicity, you just need a way to lock one or more entities (whatever the unit of atomic updates are). A CoW filesystem in direct mode (zero page caching) would need neither but could still support atomic updates to file (names). > A consistent networked API would require you to hit the metadata server for every operation. No caching. Your system would grind to a halt. Sorry, I don't mean consistent in the ACID context, I mean consistent in the loosely defined API shape context. Think NFS or 9P. I also disagree with this to some degree: pipelined operations would certainly still be possible and performant but would be rather clunky. End-to-end latency for get->update-write, the common mode of operation, would be pretty awful. > Finally, nothing in the POSIX spec prohibits an atomic filesystem or consistency guarantees. It is just that no one wants to implement these things that way because it overprovisions for one property at the expense of others. I didn't say it did, but it doesn't require it which means it effectively doesn't exist as far as the users of FS APIs are concerned. Rename operations are the only API that atomicity is required by POSIX. However without a CAS-like operation you can't safely implement a lock without several extra syscalls.
    - mrlongroots 1y ago
      I was confused for a while about where this discussion was going, and what was the broader point. Will try to consolidate thoughts in the interest of making it clearer. You seem unhappy with POSIX because its guarantees feel incomplete and ad hoc (they are). You like databases because their guarantees are more robust (also true). DBMS over POSIX enables all the guarantees that you like. I'd want to invoke the end-to-end systems argument here and say that this is how systems are supposed to work: POSIX is closer to the hardware, and as a result it is messier. It's the same reason TCP in-order guarantees are layered above the IP layer. Some of your points re: how the lower layers work seem incorrect, but that doesn't matter in the interest of the big picture. The suggestion (re: microkernels) seems to be that POSIX has a privileged position in the system stack, and that somehow prevents a stronger alternative from existing. I'd say that your gripes with POSIX may be perfectly justified, but nothing prevents a DBMS from owning a complete block device, completely circumventing the filesystem. POSIX is the default, but it is not really privileged by any means.
Ericson2314 1y ago
The filesystem interface is only privilaged interface because it is the kernel knows about. E.g. you can already use FUSE and NFS to roll your own FS implementations, but those do not a microkernel make, because the OS is still in the way dictating the implementation. The safest way to put the FS on a level-playing field with other interfaces is to make the kernel not know about, just as it doesn't know about, say, SQL.

qwertox 1y ago

I can't agree with this. I like it that I can have all these tools which work with files and are tools which are not db-oriented, and the fact that there are different filesystems for different scenarios, that I can sandwich LVM between a FS and the block device. That /proc/ can pretend to be a FS because else we'd possibly end up with something like the Windows Registry for these operations, only managed through a database. Would you store all your ~/ in something like SQLite database?

hdevalence 1y ago
Yes, I would
90s_dev 1y ago
> Would you store all your ~/ in something like SQLite database? Actually yeah that sounds pretty good. For Desktop/Finder/Explorer you'd just need a nice UI. Searching Documents/projects/etc would be the same just maybe faster? All the arbitrary stuff like ~/.npm/**/* would stop cluttering up my ls -la in ~ and could be stored in their own tables whose names I genuinely don't care about. (This was the dream of ~/Library, no?) [edit] Ooooh, I get it now. This doesn't solve namespacing or traversal.
- fc417fc802 1y ago
  > This doesn't solve namespacing or traversal. That's "just" API. FS is "just" a KV store with a weird crufty API and a few extra tricks (bind mounts or whatever). I think the primary issue is the difference in performance between different strategies. It would be interesting to have a FS with different types of folders similar to how (for example) btrfs is generally CoW but you can turn that off via an attribute.

foobiekr 1y ago

Every single time this has been tried it has gone wrong, but sure. Almost all of the operations done on actual filesystems are not database like, they are close to the underlying hardware for practical reasons. If you want a database view, add one in an upper layer.

jonhohle 1y ago
BeOS got it right with BeFS. An Email client was just a folder. MP3s could be sorted and filtered in the file system. https://news.ycombinator.com/item?id=12309686 https://news.ycombinator.com/item?id=12309686
- int_19h 1y ago
  Windows does something similar with Explorer today when you open a folder that has mostly music files in it.
- foobiekr 1y ago
  BeFS wasn't a database. It had indexed queries on EAs and they had the habit of asking application files to add their indexable content to the EAs. Internally it was just a mostly-not-transactional collection of btrees. There was no query language for updating files, or even inspecting anything about a file that was not published in the EAs (or implicitly do as with adapters), there were no multi-file transactions, no joins, nothing. Just rich metadata support in the FS.
  - Ericson2314 1y ago
    Yeah I am talking more deep architecture, and BeOS is more notable here mostly on just the user-interface level. However, I think it is reasonable to think that with way more time and money, these things would meet up. Think about it as digging a tunnel from both sides of the mountain.
    - foobiekr 1y ago
      Microsoft poured at least $100M into this hole with nothing to show for it.

01HNNWZ0MV43FF 1y ago

You could do a loopback network filesystem and make any user-space FS you want. That's what WSL does, and there's a Rust crate for it. Can't recall the name at all.

Ericson2314 1y ago
There is NFS and FUSE so you can write your own implementation, but you are still stuck with the interface that the kernel understands.

deleted 1y ago

[deleted]

runlaszlorun 1y ago

I've heard this mentioned a couple times but what would this look like functionality wise? A single "files" table with columns? Different tables for different categories of files? FTS? Something else?

Ericson2314 1y ago
See the other comments. The point is not a specific new interface, but a separation of concerns, and leveling the playing field. I'll try to do an example. The kernel doesn't currently know about SQL. Instead, you e.g. connect to a socket, and start talking to postgres. Imagine if FS stuff was the same thing: you connect to a socket, and then issue various command to read and write files. Ignore perf for a moment, it works right? Now, one counter-argument might be "hold up, what is this socket you need to connect to, isn't that part of a file system? Is there now an all-userspace inner filesystem, still kernel-supported 'meta filesystem'?" Well, the answer to that is maybe the Unix idea of making communication channels like pipes and (to a lesser extent) sockets, was a bad idea. Or rather, there may be nothing wrong with saying a directory can have a child which may be such a communication channel, but there is a problem with saying that every such communication channel should live inside some directory.

mrlongroots 1y ago

Thoughts: 1. Distributed filesystems do often use databases for metadata (FoundationDB for 3FS being a recent example) 2. Using a B+ tree for metadata is not much different from having a sorted index 3. Filesystems are a common enough usecase that skipping the abstraction complexity to co-optimize the stack is warranted