5 ms·

I think one of the problems is that there are many datasets where objects belong to multiple hierarchies and different hierarchies are more efficient for differ

I think one of the problems is that there are many datasets where objects belong to multiple hierarchies and different hierarchies are more efficient for different tasks. For example I work with medical imaging data. Typically that gets organized around the DICOM object models which even defines multiple structures. Typically that's around the patient/encounter model and slight variations of that and data is stored in a database called a PACS, but working with PACS is extremely difficult because DICOM is optimized for clinical use cases. But there are other ways of organizing the data that are more efficient for other tasks for example for quality improvement or assurance or process monitoring. In fact different users of the data are likely to want views based on different hierarchies. Some software expects certain data layouts etc. There are some efforts to standardize file hierarchies and naming for certain tasks, but perhaps you're not doing that task. You can do things like symbolic links, but trees of symbolic links end up being super fragile in my experience and they're not particularly well supported on some operating systems.

DerArzt 5y ago
Couldn't that still be stored in a traditional folder hierarchy, but all in one folder and keep track of the tags related to file names in a sqlite database?
- fluidcruft 5y ago
  I think the point is that if you're just using the filesystem as blob storage and interacting with a layer database, then you've actually moved beyond the filesystem. Just replace filenames with inodes and that's what you've done.
  - formerly_proven 5y ago
    But that's exactly what many applications have been doing for a long time, they're using S3-like services as CAS or quasi-CAS and generate views for different workflows on the fly from a separate database.
    - fluidcruft 5y ago
      Why shouldn't that functionality be more user-friendly and accessible to users?
      - DerArzt 5y ago
        It may be worth taking a step back and asking who the users are. I know that like most folks on HN, I am not your average computer user so my view is probably a bit different. With that out of the way, I can say that I firmly prefer folder hierarchies for organizing my files over tagging. I switched to MacOS when I started my new job, and the fact that tagging is such a integral feature out of the box over plain folder navigation irks me. It's easier to save a file to the correct spot in the hierarchy and find it later rather than searching by tags to me. Perhaps my view isn't consistent with a lot of computer users, but I will say that most people understand how hierarchical filing works better than tagging as one is more spatial than the other. After all, there is a reason that the memory palace technique for recall relies on imagining a physical space.
        fluidcruft 5y ago
        A hierarchy is actually a tag, so really what you're saying is you firmly believe in only having a single tag for each file and a specific adhoc schema for those tags. I think once you've used a filesystem that behaves in more complex ways you start to realize that's not necessary. For example with a snapshotting filesystem you add dates and times to the path in order to access older versions of files. I don't really use MacOS but I have noticed the tagging in my limited experience with it, which seemed reminiscent of what the old MacOS used to do where you could set icon colors. So whatever they're doing for tags isn't really the only story. MacOS is a good example though because that is a filesystem that maintains separate metadata in addition to the file contents (they used to be called resource forks, but I'm not up on modern MacOS). Many linux filesystems also can support that sort of thing. But those are just flavoring the files and not really adding different ways to index into the filesystem. A hierarchical index such as nested folders is probably a minimally viable solution and is very useful. But that doesn't mean it's not limiting and that there are not different larger solutions that achieve more useful results. Once you start thinking about snapshots and transparent de-duplication and other ideas like don't-repeat-yourself things wind up becoming far less clear.
bborud 5y ago
I don't think the filesystem is the right place to solve these types of problems. I also think if you try to make the kind of filesystems that solve these problems you'll invariably end up with entirely new problems you really don't want (complexity, performance issues, unclear semantics etc). You usually want more flexibility and control over how your data is projected to storage. (Say for instance you run out of storage and the scheme doesn't have any way to split the data across multiple filesystems). And you really want integrity constraints that stop you from pointing into thin air and help you clean up. Occasionally you also need to have the concept of identity (how do you refer to a given entity directly) without it being part of a projection that may have stopped existing - like a tag being deleted).
- fluidcruft 5y ago
  I'm not so sure about that. I don't think anyone really wants to think about that at all. I'm probably spoiled by zfs and OneDrive but I think you just have pools of space and let the filesystem take care of itself. Plug in more space and the system run a "rebalancing" or whatever, etc. Let blobs move into online storage and fetch as needed or whatever. If I want a certain set of data just tell the system to "prepare" it for use.
  - bborud 5y ago
    That supposes that you have a decent filesystem like ZFS and know how to configure it. I run ZFS and I can't even remember how to add a disk and increase the size of a storage pool without referring to the manual page or do a web search. And I only know where to look because I know I'm running ZFS. (If I'm lucky it is in the filesystem command history - I know people who have used ZFS for years who didn't even know ZFS had a command history). And what if I unplug the disk and pop it into a different machine? Of if you decide to move to Windows? I do a fair bit of photography. If it taught me one thing it was that if you design software for managing lots and lots of files that craps out if something spans a filesystem border, you'll have a truly miserable time. Photo editing software used to be like that.
    - 5y ago
theamk 5y ago
I think this is just a general lack of software for rare cases like clinical data? I mean, it sounds like what you have described is solved for photos. I use digikam photo manager, and it automatically discovers all the photos on multiple the volumes, and supports showing files by date, location, tags, path -- whatever you like. And it is not very fragile at all -- it identifies photos by metadata-excluded hash, so moving the photos around does not break the links. And back in Winamp days, it had an MP3 database which had basically the same properties. I have no doubt that users could use more and better organization, but it seems like UX problem, not a filesystem one.
- fluidcruft 5y ago
  The problem is that all the data is not the clinical data. Images are fairly well understood for clinical use, it's all the non-imaging things that need to be associated with the images. It's images and a whole bunch of stuff that's associated with them that people want to do with the data that are non-clinical. For example now there's things like BIDS[1] which are basically applying rigid schema to a filesystem. But that's only because so many people got frustrated with everyone doing their own thing and having to spend time restructuring data to different site workflows. But even that only works for neuro, what about cardiac or liver or spectroscopy or... software that doesn't use BIDS etc? And even with BIDS mostly it's just copy rename to the layouts the software you're running actually expects. There's also a huge push for "Vendor Neutral Archives" to augment PACS which will allow things to be sort of structured and managed but that always seems to be uploading and downloading from websites so people still keep copying everything out because everything is so opaque vs filesystem. > it identifies photos by metadata-excluded hash, so moving the photos around does not break the links That only solves the problem in one direction. If you run across an image after it's been moved, you know which image it is and can index into the database based on that. But if you want to find an image starting from the metadata after it's been moved, then you're stuck trawling everywhere. The metadata-hash thing exists in medical images DICOM as various GUIDs (in a different format), so you can track things and updates that using that key. But if you have to visit 30 TB of files just to find one that's been renamed or updated, it's basically impossible. [1] http://fmri.ucsd.edu/pdf/BIDS_Presentation_14NOV2018.pdf http://fmri.ucsd.edu/pdf/BIDS_Presentation_14NOV2018.pdf

DerArzt 5y ago

Couldn't that still be stored in a traditional folder hierarchy, but all in one folder and keep track of the tags related to file names in a sqlite database?

fluidcruft 5y ago
I think the point is that if you're just using the filesystem as blob storage and interacting with a layer database, then you've actually moved beyond the filesystem. Just replace filenames with inodes and that's what you've done.
- formerly_proven 5y ago
  But that's exactly what many applications have been doing for a long time, they're using S3-like services as CAS or quasi-CAS and generate views for different workflows on the fly from a separate database.
  - fluidcruft 5y ago
    Why shouldn't that functionality be more user-friendly and accessible to users?
    - DerArzt 5y ago
      It may be worth taking a step back and asking who the users are. I know that like most folks on HN, I am not your average computer user so my view is probably a bit different. With that out of the way, I can say that I firmly prefer folder hierarchies for organizing my files over tagging. I switched to MacOS when I started my new job, and the fact that tagging is such a integral feature out of the box over plain folder navigation irks me. It's easier to save a file to the correct spot in the hierarchy and find it later rather than searching by tags to me. Perhaps my view isn't consistent with a lot of computer users, but I will say that most people understand how hierarchical filing works better than tagging as one is more spatial than the other. After all, there is a reason that the memory palace technique for recall relies on imagining a physical space.
      - fluidcruft 5y ago
        A hierarchy is actually a tag, so really what you're saying is you firmly believe in only having a single tag for each file and a specific adhoc schema for those tags. I think once you've used a filesystem that behaves in more complex ways you start to realize that's not necessary. For example with a snapshotting filesystem you add dates and times to the path in order to access older versions of files. I don't really use MacOS but I have noticed the tagging in my limited experience with it, which seemed reminiscent of what the old MacOS used to do where you could set icon colors. So whatever they're doing for tags isn't really the only story. MacOS is a good example though because that is a filesystem that maintains separate metadata in addition to the file contents (they used to be called resource forks, but I'm not up on modern MacOS). Many linux filesystems also can support that sort of thing. But those are just flavoring the files and not really adding different ways to index into the filesystem. A hierarchical index such as nested folders is probably a minimally viable solution and is very useful. But that doesn't mean it's not limiting and that there are not different larger solutions that achieve more useful results. Once you start thinking about snapshots and transparent de-duplication and other ideas like don't-repeat-yourself things wind up becoming far less clear.

bborud 5y ago

I don't think the filesystem is the right place to solve these types of problems. I also think if you try to make the kind of filesystems that solve these problems you'll invariably end up with entirely new problems you really don't want (complexity, performance issues, unclear semantics etc). You usually want more flexibility and control over how your data is projected to storage. (Say for instance you run out of storage and the scheme doesn't have any way to split the data across multiple filesystems). And you really want integrity constraints that stop you from pointing into thin air and help you clean up. Occasionally you also need to have the concept of identity (how do you refer to a given entity directly) without it being part of a projection that may have stopped existing - like a tag being deleted).

fluidcruft 5y ago
I'm not so sure about that. I don't think anyone really wants to think about that at all. I'm probably spoiled by zfs and OneDrive but I think you just have pools of space and let the filesystem take care of itself. Plug in more space and the system run a "rebalancing" or whatever, etc. Let blobs move into online storage and fetch as needed or whatever. If I want a certain set of data just tell the system to "prepare" it for use.
- bborud 5y ago
  That supposes that you have a decent filesystem like ZFS and know how to configure it. I run ZFS and I can't even remember how to add a disk and increase the size of a storage pool without referring to the manual page or do a web search. And I only know where to look because I know I'm running ZFS. (If I'm lucky it is in the filesystem command history - I know people who have used ZFS for years who didn't even know ZFS had a command history). And what if I unplug the disk and pop it into a different machine? Of if you decide to move to Windows? I do a fair bit of photography. If it taught me one thing it was that if you design software for managing lots and lots of files that craps out if something spans a filesystem border, you'll have a truly miserable time. Photo editing software used to be like that.
  - 5y ago

theamk 5y ago

I think this is just a general lack of software for rare cases like clinical data? I mean, it sounds like what you have described is solved for photos. I use digikam photo manager, and it automatically discovers all the photos on multiple the volumes, and supports showing files by date, location, tags, path -- whatever you like. And it is not very fragile at all -- it identifies photos by metadata-excluded hash, so moving the photos around does not break the links. And back in Winamp days, it had an MP3 database which had basically the same properties. I have no doubt that users could use more and better organization, but it seems like UX problem, not a filesystem one.

fluidcruft 5y ago
The problem is that all the data is not the clinical data. Images are fairly well understood for clinical use, it's all the non-imaging things that need to be associated with the images. It's images and a whole bunch of stuff that's associated with them that people want to do with the data that are non-clinical. For example now there's things like BIDS[1] which are basically applying rigid schema to a filesystem. But that's only because so many people got frustrated with everyone doing their own thing and having to spend time restructuring data to different site workflows. But even that only works for neuro, what about cardiac or liver or spectroscopy or... software that doesn't use BIDS etc? And even with BIDS mostly it's just copy rename to the layouts the software you're running actually expects. There's also a huge push for "Vendor Neutral Archives" to augment PACS which will allow things to be sort of structured and managed but that always seems to be uploading and downloading from websites so people still keep copying everything out because everything is so opaque vs filesystem. > it identifies photos by metadata-excluded hash, so moving the photos around does not break the links That only solves the problem in one direction. If you run across an image after it's been moved, you know which image it is and can index into the database based on that. But if you want to find an image starting from the metadata after it's been moved, then you're stuck trawling everywhere. The metadata-hash thing exists in medical images DICOM as various GUIDs (in a different format), so you can track things and updates that using that key. But if you have to visit 30 TB of files just to find one that's been renamed or updated, it's basically impossible. [1] http://fmri.ucsd.edu/pdf/BIDS_Presentation_14NOV2018.pdf http://fmri.ucsd.edu/pdf/BIDS_Presentation_14NOV2018.pdf