3 ms·

I absolutely agree that we need to move away from the idea of physical folders. The file hierarchy IS very useful way of categorizing your data, but a file shou

I absolutely agree that we need to move away from the idea of physical folders. The file hierarchy IS very useful way of categorizing your data, but a file should be able to belong to multiple categories. That's why we have hard links, but I digress. I wrote an emacs package called SFS(search file system) to solve exactly this set of problems. It is just an interface on top of the excellent Recoll full-text indexer. Among other things, SFS allows you to create hierarchies of queries, the search analog of a folder hierarchy. A parent "directory" is just the logical OR of all of it's content named queries. Everything works great for me, but the installation process is a bit painful(especially cross-platform), and indexing can take a serious amount of time and space initially, so I would say it's still a ways from being really comfy to use. But it is totally possible to have your own little google for your file system. I have gifs on the GitHub page to at least give you an idea: https://github.com/Overdr0ne/sfs https://github.com/Overdr0ne/sfs Enjoy

dredmorbius 5y ago
Hard links prove quite fragile (many tools will remove the link then replace it with a new unlinked file) and occasionally dangerous. Tagging may work better, though you'd need a tags-aware toolchain to work with them if the metadata are associated with the filesystem. An external tagstore might be resilient but could find itself out-of-sync with filesystem state.
- Overdr0ne 5y ago
  Hmm, do you have some example of these dangerous scenarios? I suspect a lot of tools have come to think of the "file path" as the identifier for a file. So of course they would break if that classification were to be reorganized, like if two parent categories were to swap. I would call that an abuse of the file system though. But the status quo is what it is. Most file systems already have lots of other metadata built in that can be used to access the inode or whatever you call your data structure. My point is, accessing data in a more general case is a search operation. As far as an external tag store, that is basically what a search index is. And Recoll is full-text, so each file has a shit-ton of tags associated with it. You then just pass the -m flag to the indexer, and it monitors for file modifications, and updates the index accordingly. I have not noticed a significant performance impact there. Mostly just the initial index operation sucks.
  - dredmorbius 5y ago
    Some I know of, some I'm presuming, and there are all but certainly others. Hardlinked directories create all kinds of mischeif. That's the principle issue. It's often entirely disabled. Recursive directory trees are all kinds of fun. (Moreso than even the symlinked version.) Given a hardlink exists, a tool which operates by 1) removing the file (deletes the local directory entry to the hardlink inode), 2) creates a new file (same name, new inode, not hardlinked), and then 3) populates that with new content, creates the issue of a presumed identical hardlink existing where that's not the case. Hardlinks with relative directory references will reference different files, or configurations, or executables, or devices, from different points on the filesystem. ... or within different filesystem chroots. Hardlinks might be used to break out of a chroot or similar jail. A process which could change the hardlink could affect other processes outside the jail. As for tags: These are ... generally ... not the same as what most people have in mind as a full-text index, or at the very least, a special class of index. I'm thinking of a controlled-vocabulary generally instantiated as an RDF triple, though folksononmies and casual tagging systems are also often used. The problem occurs when you've got a tagged data store that's being modified by non-tag-aware tools. There are reasons why that might be permitted and/or necessary, though also problematic. My sense is that robust tagging probably needs implementing at the filesystem level.
    - Overdr0ne 5y ago
      Yeah, not a fan of recursive directory trees. Sysfs for example is pretty wonky esp when you're searching for some specific attribute of the device. Not hard links or real files ftm, but same idea. Hard linked directories breaks the category system. Now the same name for a different inode in two directories is a point well taken, but I would argue that does not fully describe the inode, that name is just one component of the metadata for that file. People are just so unaware of all that other metadata because the interface rarely shows it to them. So many people have taken to packing all that data into the filename. Version numbers, code names- it's one way to achieve portability i guess, but what an ugly compromise! And with all the virtual environments now for pythons and such, it's quite easy to find yourself using the wrong version of something if you don't really know what you're doing and just look at the filename. Hard linking links all that metadata, which of course does include that unique ID that open returns, so I think it's okay. I would just like to see our file interfaces more adapted to showing all that important metadata in a comfier way

dredmorbius 5y ago

Hard links prove quite fragile (many tools will remove the link then replace it with a new unlinked file) and occasionally dangerous. Tagging may work better, though you'd need a tags-aware toolchain to work with them if the metadata are associated with the filesystem. An external tagstore might be resilient but could find itself out-of-sync with filesystem state.

Overdr0ne 5y ago
Hmm, do you have some example of these dangerous scenarios? I suspect a lot of tools have come to think of the "file path" as the identifier for a file. So of course they would break if that classification were to be reorganized, like if two parent categories were to swap. I would call that an abuse of the file system though. But the status quo is what it is. Most file systems already have lots of other metadata built in that can be used to access the inode or whatever you call your data structure. My point is, accessing data in a more general case is a search operation. As far as an external tag store, that is basically what a search index is. And Recoll is full-text, so each file has a shit-ton of tags associated with it. You then just pass the -m flag to the indexer, and it monitors for file modifications, and updates the index accordingly. I have not noticed a significant performance impact there. Mostly just the initial index operation sucks.
- dredmorbius 5y ago
  Some I know of, some I'm presuming, and there are all but certainly others. Hardlinked directories create all kinds of mischeif. That's the principle issue. It's often entirely disabled. Recursive directory trees are all kinds of fun. (Moreso than even the symlinked version.) Given a hardlink exists, a tool which operates by 1) removing the file (deletes the local directory entry to the hardlink inode), 2) creates a new file (same name, new inode, not hardlinked), and then 3) populates that with new content, creates the issue of a presumed identical hardlink existing where that's not the case. Hardlinks with relative directory references will reference different files, or configurations, or executables, or devices, from different points on the filesystem. ... or within different filesystem chroots. Hardlinks might be used to break out of a chroot or similar jail. A process which could change the hardlink could affect other processes outside the jail. As for tags: These are ... generally ... not the same as what most people have in mind as a full-text index, or at the very least, a special class of index. I'm thinking of a controlled-vocabulary generally instantiated as an RDF triple, though folksononmies and casual tagging systems are also often used. The problem occurs when you've got a tagged data store that's being modified by non-tag-aware tools. There are reasons why that might be permitted and/or necessary, though also problematic. My sense is that robust tagging probably needs implementing at the filesystem level.
  - Overdr0ne 5y ago
    Yeah, not a fan of recursive directory trees. Sysfs for example is pretty wonky esp when you're searching for some specific attribute of the device. Not hard links or real files ftm, but same idea. Hard linked directories breaks the category system. Now the same name for a different inode in two directories is a point well taken, but I would argue that does not fully describe the inode, that name is just one component of the metadata for that file. People are just so unaware of all that other metadata because the interface rarely shows it to them. So many people have taken to packing all that data into the filename. Version numbers, code names- it's one way to achieve portability i guess, but what an ugly compromise! And with all the virtual environments now for pythons and such, it's quite easy to find yourself using the wrong version of something if you don't really know what you're doing and just look at the filename. Hard linking links all that metadata, which of course does include that unique ID that open returns, so I think it's okay. I would just like to see our file interfaces more adapted to showing all that important metadata in a comfier way