5 ms·

Show HN: Reimagine the File System

49 points by daenz 6y ago

Shared404 6y ago
I don't know enough about filesystems to speak as to whether it's a good or bad design, but I quite like the concept. Cool work!
- eyelidlessness 6y ago
  It’s a great design for certain use cases and fundamentally incompatible with/needs adaptation for others. It’ll work well for: - Aggregate workflows (i.e. composing heterogeneous collections) - Library curation (i.e. individual or mass categorization) - Content discovery/rediscovery (i.e. find things based on metadata rather than location) It’ll either poorly serve or need additional consideration for: - Contained dependencies (i.e. any kind of encapsulated package that has internal references) - Structural authorization (i.e. a given user may or may not be permitted access to some or all functions based on the branch/leaf they’re trying to access) - Structural layout (i.e. the user has a hierarchy in mind which they wish to reify)
  - eyelidlessness 6y ago
    I’ll leave the above comment intact from here but I already edited once and still regret the word choice “fundamentally incompatible”. As with any data structure, you can make adaptations and accommodations to support use cases that seem to conflict. There’s always a memory/performance/impedance mismatch/experience cost for any abstraction, but there’s no reason a set/tree hybrid couldn’t provide for both families of use cases, given those costs.
sanketsarang 6y ago
Have you performance benchmarked this? Would be interested in giving it a try if performance is not compromised. Currently use XFS.
- daenz 6y ago
  Not formally, no, but anecdotally, I do notice some performance impact with with many files and tags, and some of the more aggressive file browsers like to depth-first cache directories, which can be involved when directories are generated dynamically and extremely deep. I am actively looking for solutions to that! Fortunately you can try Supertag out without touching your original files, as Supertag will use symlinks to your original files, so you don't need to move them.
- eyelidlessness 6y ago
  I can’t speak to the implementation but in principle, a filesystem essentially being a specialized database, there’s no reason techniques commonly used for SQL optimization couldn’t be applied here. It’s fundamentally set theory, where you’re obtaining your results by producing subsets rather than traversing a hierarchical tree of branches and end nodes. Of course that implies some pathological conditions (some superset/subset scenarios will produce too many results to be useful or to optimize resolution) which would produce poor UX, but sane defaults and heuristics could alleviate that. Again not speaking to the project itself, but in principle achieving good performance on something like this would boil down to “do what SQL databases do, then design efficient schemas and queries for the use case”.
- renox 6y ago
  You should only use raw block number instead of filesystems this way you would get even better performance /S
lapinot 6y ago
Neat. As a remark, this is how LDAP is structured: entries (files) are "tagged" by attributes (directories) and you access them with the same pseudo-paths which are actually conjunctions of filters. I must say honestly i'm not so fond of using ordered sequences (posix paths) in place of the unordered sets which they should be. I guess my question is really why you wanted to make a fuse fs out of it (versus say some tag-based filtering script like notmuch). (just for fun is surely a valid answer)
- Spivak 6y ago
  Because by making it a regular fs you immediately get support in every piece of software. Every file chooser on your system suddenly knows about tags and gives you a way to query them.
- daenz 6y ago
  Basically what Spivak said. I was interested in adapting an existing interface (a filesystem) to do the set intersections because it means more software could benefit from it without additional work.
seg_lol 6y ago
The Disco MapReduce framework has a tag based distributed file system. https://github.com/discoproject/discodb https://github.com/discoproject/discodb
al3xandre 6y ago
This is basically to files, what gmail has done to emails. (If you can remember emails before the gmail era)
- huhtenberg 6y ago
  Opera Mail did that too, and before GMail, I think. A single email database with tag-based filtered views on the UI end.

49 points by daenz 6y ago

I don't know enough about filesystems to speak as to whether it's a good or bad design, but I quite like the concept. Cool work!

eyelidlessness 6y ago
It’s a great design for certain use cases and fundamentally incompatible with/needs adaptation for others. It’ll work well for: - Aggregate workflows (i.e. composing heterogeneous collections) - Library curation (i.e. individual or mass categorization) - Content discovery/rediscovery (i.e. find things based on metadata rather than location) It’ll either poorly serve or need additional consideration for: - Contained dependencies (i.e. any kind of encapsulated package that has internal references) - Structural authorization (i.e. a given user may or may not be permitted access to some or all functions based on the branch/leaf they’re trying to access) - Structural layout (i.e. the user has a hierarchy in mind which they wish to reify)
- eyelidlessness 6y ago
  I’ll leave the above comment intact from here but I already edited once and still regret the word choice “fundamentally incompatible”. As with any data structure, you can make adaptations and accommodations to support use cases that seem to conflict. There’s always a memory/performance/impedance mismatch/experience cost for any abstraction, but there’s no reason a set/tree hybrid couldn’t provide for both families of use cases, given those costs.

sanketsarang 6y ago

Have you performance benchmarked this? Would be interested in giving it a try if performance is not compromised. Currently use XFS.

daenz 6y ago
Not formally, no, but anecdotally, I do notice some performance impact with with many files and tags, and some of the more aggressive file browsers like to depth-first cache directories, which can be involved when directories are generated dynamically and extremely deep. I am actively looking for solutions to that! Fortunately you can try Supertag out without touching your original files, as Supertag will use symlinks to your original files, so you don't need to move them.
eyelidlessness 6y ago
I can’t speak to the implementation but in principle, a filesystem essentially being a specialized database, there’s no reason techniques commonly used for SQL optimization couldn’t be applied here. It’s fundamentally set theory, where you’re obtaining your results by producing subsets rather than traversing a hierarchical tree of branches and end nodes. Of course that implies some pathological conditions (some superset/subset scenarios will produce too many results to be useful or to optimize resolution) which would produce poor UX, but sane defaults and heuristics could alleviate that. Again not speaking to the project itself, but in principle achieving good performance on something like this would boil down to “do what SQL databases do, then design efficient schemas and queries for the use case”.
renox 6y ago
You should only use raw block number instead of filesystems this way you would get even better performance /S

lapinot 6y ago

Neat. As a remark, this is how LDAP is structured: entries (files) are "tagged" by attributes (directories) and you access them with the same pseudo-paths which are actually conjunctions of filters. I must say honestly i'm not so fond of using ordered sequences (posix paths) in place of the unordered sets which they should be. I guess my question is really why you wanted to make a fuse fs out of it (versus say some tag-based filtering script like notmuch). (just for fun is surely a valid answer)

Spivak 6y ago
Because by making it a regular fs you immediately get support in every piece of software. Every file chooser on your system suddenly knows about tags and gives you a way to query them.
daenz 6y ago
Basically what Spivak said. I was interested in adapting an existing interface (a filesystem) to do the set intersections because it means more software could benefit from it without additional work.

seg_lol 6y ago

The Disco MapReduce framework has a tag based distributed file system. https://github.com/discoproject/discodb https://github.com/discoproject/discodb

al3xandre 6y ago

This is basically to files, what gmail has done to emails. (If you can remember emails before the gmail era)

huhtenberg 6y ago
Opera Mail did that too, and before GMail, I think. A single email database with tag-based filtered views on the UI end.