3 ms·

I don't think the filesystem is the right place to solve these types of problems. I also think if you try to make the kind of filesystems that solve these probl

by bborud 5y ago

I don't think the filesystem is the right place to solve these types of problems. I also think if you try to make the kind of filesystems that solve these problems you'll invariably end up with entirely new problems you really don't want (complexity, performance issues, unclear semantics etc). You usually want more flexibility and control over how your data is projected to storage. (Say for instance you run out of storage and the scheme doesn't have any way to split the data across multiple filesystems). And you really want integrity constraints that stop you from pointing into thin air and help you clean up. Occasionally you also need to have the concept of identity (how do you refer to a given entity directly) without it being part of a projection that may have stopped existing - like a tag being deleted).

fluidcruft 5y ago
I'm not so sure about that. I don't think anyone really wants to think about that at all. I'm probably spoiled by zfs and OneDrive but I think you just have pools of space and let the filesystem take care of itself. Plug in more space and the system run a "rebalancing" or whatever, etc. Let blobs move into online storage and fetch as needed or whatever. If I want a certain set of data just tell the system to "prepare" it for use.
- bborud 5y ago
  That supposes that you have a decent filesystem like ZFS and know how to configure it. I run ZFS and I can't even remember how to add a disk and increase the size of a storage pool without referring to the manual page or do a web search. And I only know where to look because I know I'm running ZFS. (If I'm lucky it is in the filesystem command history - I know people who have used ZFS for years who didn't even know ZFS had a command history). And what if I unplug the disk and pop it into a different machine? Of if you decide to move to Windows? I do a fair bit of photography. If it taught me one thing it was that if you design software for managing lots and lots of files that craps out if something spans a filesystem border, you'll have a truly miserable time. Photo editing software used to be like that.
  - fluidcruft 5y ago
    I actually use Windows (and OneDrive) quite a lot nowadays. I have very little problem thinking of disk space as a sort of OneDrive cache at this point and basically you have different devices that are faster vs slower. And when that's the case detaching a disk just means removing a copy of the data. But datasets in general on ZFS are great because you can just mount and unmount them at will. They're there on the disk, but if you don't need them they're not mounted. This is great if you don't want to accidentally modify some subset of data you're not working with. One of my favorite features of ZFS are the encrypted datasets where you can have chunks of data stored with different encryption on the sane disk and all the integrity and migration etc works without needing to decrypt anything. Which is also great because it means ransomware can't touch it. I do think it's useful to have some sort of "chunking" of the data like that which mostly maps to ownership. This data belongs to this client/project, that data belongs to that client/project, etc. Often those divisions have different use restrictions so I've found it great to isolate things that way. And again ZFS is great at this because you can just dump out the full encrypted datasets for handoff/archival or whatever. But anyway I would point out that filesystem boundaries come from whatever the filesystem implements and exposes to the application. The only real issue with filesystem boundaries is that renames that move files between devices aren't possible. And that's an artifact of that filesystem's design needing to synchronize the hierarchy at a hardware level. If you could call up the same path and it comes from wherever it happens to be then it's not a problem. Like in OneDrive when you open a file that's only online and it needs to suck it down first.
kaba0 5y ago
I completely agree on the constraints you mention, but I don’t actually see how these can be solved without the knowledge of the file system. Let’s say you want to create a tagging system on top of a traditional file system. Let’s say you create a folder for a tag and make it have a symlink to each entry with that tag. Now any single move, delete operation will render your tag lib incorrect and there is no cheap way to correct it at each change.

by bborud 5y ago

fluidcruft 5y ago

I'm not so sure about that. I don't think anyone really wants to think about that at all. I'm probably spoiled by zfs and OneDrive but I think you just have pools of space and let the filesystem take care of itself. Plug in more space and the system run a "rebalancing" or whatever, etc. Let blobs move into online storage and fetch as needed or whatever. If I want a certain set of data just tell the system to "prepare" it for use.

bborud 5y ago
That supposes that you have a decent filesystem like ZFS and know how to configure it. I run ZFS and I can't even remember how to add a disk and increase the size of a storage pool without referring to the manual page or do a web search. And I only know where to look because I know I'm running ZFS. (If I'm lucky it is in the filesystem command history - I know people who have used ZFS for years who didn't even know ZFS had a command history). And what if I unplug the disk and pop it into a different machine? Of if you decide to move to Windows? I do a fair bit of photography. If it taught me one thing it was that if you design software for managing lots and lots of files that craps out if something spans a filesystem border, you'll have a truly miserable time. Photo editing software used to be like that.
- fluidcruft 5y ago
  I actually use Windows (and OneDrive) quite a lot nowadays. I have very little problem thinking of disk space as a sort of OneDrive cache at this point and basically you have different devices that are faster vs slower. And when that's the case detaching a disk just means removing a copy of the data. But datasets in general on ZFS are great because you can just mount and unmount them at will. They're there on the disk, but if you don't need them they're not mounted. This is great if you don't want to accidentally modify some subset of data you're not working with. One of my favorite features of ZFS are the encrypted datasets where you can have chunks of data stored with different encryption on the sane disk and all the integrity and migration etc works without needing to decrypt anything. Which is also great because it means ransomware can't touch it. I do think it's useful to have some sort of "chunking" of the data like that which mostly maps to ownership. This data belongs to this client/project, that data belongs to that client/project, etc. Often those divisions have different use restrictions so I've found it great to isolate things that way. And again ZFS is great at this because you can just dump out the full encrypted datasets for handoff/archival or whatever. But anyway I would point out that filesystem boundaries come from whatever the filesystem implements and exposes to the application. The only real issue with filesystem boundaries is that renames that move files between devices aren't possible. And that's an artifact of that filesystem's design needing to synchronize the hierarchy at a hardware level. If you could call up the same path and it comes from wherever it happens to be then it's not a problem. Like in OneDrive when you open a file that's only online and it needs to suck it down first.

kaba0 5y ago

I completely agree on the constraints you mention, but I don’t actually see how these can be solved without the knowledge of the file system. Let’s say you want to create a tagging system on top of a traditional file system. Let’s say you create a folder for a tag and make it have a symlink to each entry with that tag. Now any single move, delete operation will render your tag lib incorrect and there is no cheap way to correct it at each change.