3 ms·

A virtual file system? How usable are those? I want something that can store slightly different data (modifies data on the fly), and one, that I can use as a ta

by bmacho 3y ago

A virtual file system? How usable are those? I want something that can store slightly different data (modifies data on the fly), and one, that I can use as a tagged file system (generates folder structure on the fly). Is that viable with FUSE/NFS? Do programs play nice with them (no caching, no recursive lookahead and such)? Is it hard to write one? Which language did you used?

eru 3y ago
I wrote version in Python and Rust. (And ended up contributing to libfuse and its Rust equivalent along the way.) > Do programs play nice with them (no caching, no recursive lookahead and such)? Other programs play nice with my peculiar file systems as far as I can tell: they just look like ordinary files and directories and symlinks etc to them. I don't know how well that would work for your usecase. Most of the caching etc that I am aware of happens at the layer of the Linux kernel, and your FUSE daemon can configure what caching it wants or doesn't want. I don't know much about NFS. I only used FUSE directly. For me, presenting git as a filesystem was relatively easy to write. I think that comes down to two main factors: * Git's internal structure is very close to how Linux (and especially fuse) do filesystems. That's probably not a big surprise, given that git and the Linux kernel are both projects started by the same guy. * I exposed the git data as a read-only filesystem. In general, if you can avoid mutation, you can avoid a lot of complexity. (The main source of complexity in the Python version I wrote first was that we allowed wanted to reflect changes in the contents of git branches to be reflected in the filesystem side. That's mutability flowing in only one direction (from git to the filesystem), but it was already annoying enough to deal with.) On a more general note: I see filesystems mostly as one peculiar interface you can put in front of your data. So one some level git has an underlying database, and we just expose it. The same could be done for eg postgres or mongodb with a suitable fuse daemon. But you can also think of ext4 being such a database with a weird UI layer on top. The main difference in practice is that they run their 'fuse daemon' in the kernel. From what I've read the new bcachefs explicitly embraces this view of 'filesystems are just databases', and uses techniques borrowed from relational databases for its internal datastructures.

by bmacho 3y ago

eru 3y ago

I wrote version in Python and Rust. (And ended up contributing to libfuse and its Rust equivalent along the way.) > Do programs play nice with them (no caching, no recursive lookahead and such)? Other programs play nice with my peculiar file systems as far as I can tell: they just look like ordinary files and directories and symlinks etc to them. I don't know how well that would work for your usecase. Most of the caching etc that I am aware of happens at the layer of the Linux kernel, and your FUSE daemon can configure what caching it wants or doesn't want. I don't know much about NFS. I only used FUSE directly. For me, presenting git as a filesystem was relatively easy to write. I think that comes down to two main factors: * Git's internal structure is very close to how Linux (and especially fuse) do filesystems. That's probably not a big surprise, given that git and the Linux kernel are both projects started by the same guy. * I exposed the git data as a read-only filesystem. In general, if you can avoid mutation, you can avoid a lot of complexity. (The main source of complexity in the Python version I wrote first was that we allowed wanted to reflect changes in the contents of git branches to be reflected in the filesystem side. That's mutability flowing in only one direction (from git to the filesystem), but it was already annoying enough to deal with.) On a more general note: I see filesystems mostly as one peculiar interface you can put in front of your data. So one some level git has an underlying database, and we just expose it. The same could be done for eg postgres or mongodb with a suitable fuse daemon. But you can also think of ext4 being such a database with a weird UI layer on top. The main difference in practice is that they run their 'fuse daemon' in the kernel. From what I've read the new bcachefs explicitly embraces this view of 'filesystems are just databases', and uses techniques borrowed from relational databases for its internal datastructures.