5 ms·

Tagsistant: A Semantic Filesystem for Linux

105 points by pcr910303 7y ago

miohtama 7y ago
This is an interesting concept. But why to do this on a file system level? Web and desktop based applications like Google Photos and iTunes offers some cataloging and search capabilities that should cover most of the use cases. Would this later be connected to something like Spotlight on OSX?
- yjftsjthsd-h 7y ago
  One advantage to a FS over a program is that it gets to be really interoperable for free; pretty much everything can work on raw files, and the unix ecosystem is very good at inter-operating with filesystems as an "API".
- JadeNB 7y ago
  Isn't this like asking "why do this in a uniform way, when each application could do it in its own way?" (Also, (1) even if "should cover most of the use cases" is correct, why not cover all of the use cases?; and (2) the mere presence of more powerful tools can encourage ingenuity to use them in ways that wouldn't have been imagined if we only had purpose-built, specialised tools.)
- Nerada 7y ago
  For me it's simply that I'd much rather not give up my photographs to Google.
- TeMPOraL 7y ago
  Because (select formats of) photos and music aren't the only things that people want tagged. If I want to tag RAW pictures, or TV shows, or Word documents, or assets for a game, or notes, or ...., neither Google Photos nor iTunes can help me. Moreover, such a tagging system belongs to an OS-wide service; it's not something that should be implemented on its own by every application dealing with particular media types.
jlrubin 7y ago
This is really cool, I've been thinking about making something like this myself for a while! Design seems to be relatively well thought out. I might be a bit too paranoid of file loss to install it though...
- mruts 7y ago
  Then just use it with symbolic links?
nerdponx 7y ago
Interesting timing. I came across TMSU [0] the other day, which seems to have similar objectives. [0]: https://tmsu.org/ https://tmsu.org/
- rasengan0 7y ago
  Yayy! Thank you for the reference, tag my shit up seems more straightforward to me after reading tmsu and Tagsistant. and both use sqlite!
fao_ 7y ago
I was worried, because I'm currently working on a tagging system and they beat me to the release, punchy website included. Then I realised that they built it on top of FUSE, and SQL, and took a sigh of relief. ((EDIT: Perhaps this is a little harsh? I didn't mean to be harsh, just precise, but perhaps I went a little over the top -- I apologize to the authors if I did.)) I investigated the FUSE/db option earlier a year or two ago, and personally I don't see this as an interesting or compelling solution to the file<->tag problem. Because users move and rename files, pathnames are potentially semantically meaningless to the tag system. The contents of files change often, and arbitrarily (given things like MS Word's formats which are literal memory dumps of what word is doing, not to factor in encrypted files, etc.), because of this, file hashes are potentially semantically meaningless to the tag system. In other words, basic file operations (reading/writing/renaming) will cause this system to break your tags without significant work to keep the file<->tag relation consistent. You can attempt to mitigate this problem through systems that keep track of files (inotify, etc.) but that introduces a runtime cost and has technical difficulties as well. It's 'designed' (albeit unintentionally) to break from the start, and the developer has to exert a large amount of effort to stop the system from breaking. To me it didn't seem like the effort was worth it, that the innate flaws were not worth surmounting. Unfortunately to avoid this from being a 'debbie downer' post, I'd have to talk about the alternative approach, which I don't really have space (or the time, right now) to do here.
- curioussavage 7y ago
  So when is your solution going to be available?
  - fao_ 7y ago
    In short: when it's ready. I struggled with some health stuff last year which led to a delay (and also got side-tracked with some other projects that went nowhere). At the moment I'm satisfied the core library (libkoios -- koios being the greek word for 'to query' c:) works but I don't really trust myself not to have forgotten something. At the moment the program's interface has been worked through, the documentation written up, and I'm just plugging the user-level interface together, so maybe a few days to a week to run through everything. I'm still not comfortable with my application-level testing either so we'll see I guess.
xixixao 7y ago
If you find this interesting, check out Bear with its nested tags, which are executed brilliantly.
joshu 7y ago
and of course there's the requisite tag cloud
teddyh 7y ago
Isn’t stuff like this what the ext4 “user_xattr” mount option is for?
- tgbugs 7y ago
  Not quite. xattrs are not indexed, so if you want efficient retrieval by xattr tags you have to build/maintain your own inverted index. xattrs are good if you have a file that you want to store metadata about, but not so good if you want to use that metadata for discovery (at least in my experience, maybe I'm missing something, but I've implemented a remote file syncing system using xattrs and I never came across a tool in the unix arsenal beyond find + getfattr which is painfully slow).
  - zmix 7y ago
    I made heavy use of OpenMeta (that was a community effort on OSX, before Apple introduced something very similar), that placed tags in xattrs. It is great! Retrieving never caused any issues or slowdowns.
zmix 7y ago
Why not just make a macOS compatible system, based on xattrs?
jmiskovic 7y ago
Very nice project. I'm delighted to see new ideas for organizing files and personal data. I feel that most people gave up on organizing data and just went with concept of searching. This is a shame because searching wastes everyone's time to filter false positives, while small effort of tagging new content goes long way to enable discoverability. Web sites that allow you to filter content based on desired and undesired tags give you optimal way to recover information. A very interesting feature in Tagsistant is tag relations. It enables tree hierarchies for tags ("anything starwars-related is also scifi-related"). Kind of ironic how they wanted to get away from tree structure of files, and then they implemented tree structure for tags. Perhaps a tagging system for organizing your tags would be better? :) This meta-structure of data is fascinating. Is there a good resource that systematizes the area, with best practices and implementation tips?
- feanaro 7y ago
  Tree hierarchies for tags is starting to sound a lot like categories. It's probably essential not to overdo it because then you start to lose the power of tags and gain the detriments of categories. That said, I do think that introducing some limited relations between tags can be beneficial.
theon144 7y ago
Would also like to point out TMSU [0] which also presents a filesystem-y interface, alongside a quite powerful CLI. [0]: https://tmsu.org/ https://tmsu.org/
- Fnoord 7y ago
  Can it import from current file magic? ID3? CDDB? IMDB? You see, it is a big effort to manually input all this data. What is useful is reliable, public source entries. You also want a documented data format for future compatibility. So you can migrate to the next format. This furthermore gives the advantage that applications can use the metadata. Although I suppose the file system abstraction achieves the same?
amelius 7y ago
I think tags are a dead end because few users will make the effort to tag their files properly, at all times. Better use the Google approach, and let ML/NLP do the job.
- eitland 7y ago
  Users will not give their files proper names or put them into folders or tidy their desks or do the laundry either. That is some users won't do that, but by adapting everything to these users we are making everything worse for everyone including them.
  - amelius 7y ago
    The problem is that everyone is one of these users at some point. E.g. when about to leave the office and receiving an attachment. Or at 3am when solving a tough problem. Instead of requiring every file to be properly tagged, why not take the Google-approach altogether?
    - eitland 7y ago
      Perfect example. However: I'm not trying to argue against search, I'm arguing against everyone having to rely on search because ux designers decided to remove every other way of finding files because someone might someday forget to tag the file or whatever the procedure is.

105 points by pcr910303 7y ago

This is an interesting concept. But why to do this on a file system level? Web and desktop based applications like Google Photos and iTunes offers some cataloging and search capabilities that should cover most of the use cases. Would this later be connected to something like Spotlight on OSX?

yjftsjthsd-h 7y ago
One advantage to a FS over a program is that it gets to be really interoperable for free; pretty much everything can work on raw files, and the unix ecosystem is very good at inter-operating with filesystems as an "API".
JadeNB 7y ago
Isn't this like asking "why do this in a uniform way, when each application could do it in its own way?" (Also, (1) even if "should cover most of the use cases" is correct, why not cover all of the use cases?; and (2) the mere presence of more powerful tools can encourage ingenuity to use them in ways that wouldn't have been imagined if we only had purpose-built, specialised tools.)
Nerada 7y ago
For me it's simply that I'd much rather not give up my photographs to Google.
TeMPOraL 7y ago
Because (select formats of) photos and music aren't the only things that people want tagged. If I want to tag RAW pictures, or TV shows, or Word documents, or assets for a game, or notes, or ...., neither Google Photos nor iTunes can help me. Moreover, such a tagging system belongs to an OS-wide service; it's not something that should be implemented on its own by every application dealing with particular media types.

jlrubin 7y ago

This is really cool, I've been thinking about making something like this myself for a while! Design seems to be relatively well thought out. I might be a bit too paranoid of file loss to install it though...

mruts 7y ago
Then just use it with symbolic links?

nerdponx 7y ago

Interesting timing. I came across TMSU [0] the other day, which seems to have similar objectives. [0]: https://tmsu.org/ https://tmsu.org/

rasengan0 7y ago
Yayy! Thank you for the reference, tag my shit up seems more straightforward to me after reading tmsu and Tagsistant. and both use sqlite!

fao_ 7y ago

I was worried, because I'm currently working on a tagging system and they beat me to the release, punchy website included. Then I realised that they built it on top of FUSE, and SQL, and took a sigh of relief. ((EDIT: Perhaps this is a little harsh? I didn't mean to be harsh, just precise, but perhaps I went a little over the top -- I apologize to the authors if I did.)) I investigated the FUSE/db option earlier a year or two ago, and personally I don't see this as an interesting or compelling solution to the file<->tag problem. Because users move and rename files, pathnames are potentially semantically meaningless to the tag system. The contents of files change often, and arbitrarily (given things like MS Word's formats which are literal memory dumps of what word is doing, not to factor in encrypted files, etc.), because of this, file hashes are potentially semantically meaningless to the tag system. In other words, basic file operations (reading/writing/renaming) will cause this system to break your tags without significant work to keep the file<->tag relation consistent. You can attempt to mitigate this problem through systems that keep track of files (inotify, etc.) but that introduces a runtime cost and has technical difficulties as well. It's 'designed' (albeit unintentionally) to break from the start, and the developer has to exert a large amount of effort to stop the system from breaking. To me it didn't seem like the effort was worth it, that the innate flaws were not worth surmounting. Unfortunately to avoid this from being a 'debbie downer' post, I'd have to talk about the alternative approach, which I don't really have space (or the time, right now) to do here.

curioussavage 7y ago
So when is your solution going to be available?
- fao_ 7y ago
  In short: when it's ready. I struggled with some health stuff last year which led to a delay (and also got side-tracked with some other projects that went nowhere). At the moment I'm satisfied the core library (libkoios -- koios being the greek word for 'to query' c:) works but I don't really trust myself not to have forgotten something. At the moment the program's interface has been worked through, the documentation written up, and I'm just plugging the user-level interface together, so maybe a few days to a week to run through everything. I'm still not comfortable with my application-level testing either so we'll see I guess.

xixixao 7y ago

If you find this interesting, check out Bear with its nested tags, which are executed brilliantly.

joshu 7y ago

and of course there's the requisite tag cloud

teddyh 7y ago

Isn’t stuff like this what the ext4 “user_xattr” mount option is for?

tgbugs 7y ago
Not quite. xattrs are not indexed, so if you want efficient retrieval by xattr tags you have to build/maintain your own inverted index. xattrs are good if you have a file that you want to store metadata about, but not so good if you want to use that metadata for discovery (at least in my experience, maybe I'm missing something, but I've implemented a remote file syncing system using xattrs and I never came across a tool in the unix arsenal beyond find + getfattr which is painfully slow).
- zmix 7y ago
  I made heavy use of OpenMeta (that was a community effort on OSX, before Apple introduced something very similar), that placed tags in xattrs. It is great! Retrieving never caused any issues or slowdowns.

zmix 7y ago

Why not just make a macOS compatible system, based on xattrs?

jmiskovic 7y ago

Very nice project. I'm delighted to see new ideas for organizing files and personal data. I feel that most people gave up on organizing data and just went with concept of searching. This is a shame because searching wastes everyone's time to filter false positives, while small effort of tagging new content goes long way to enable discoverability. Web sites that allow you to filter content based on desired and undesired tags give you optimal way to recover information. A very interesting feature in Tagsistant is tag relations. It enables tree hierarchies for tags ("anything starwars-related is also scifi-related"). Kind of ironic how they wanted to get away from tree structure of files, and then they implemented tree structure for tags. Perhaps a tagging system for organizing your tags would be better? :) This meta-structure of data is fascinating. Is there a good resource that systematizes the area, with best practices and implementation tips?

feanaro 7y ago
Tree hierarchies for tags is starting to sound a lot like categories. It's probably essential not to overdo it because then you start to lose the power of tags and gain the detriments of categories. That said, I do think that introducing some limited relations between tags can be beneficial.

theon144 7y ago

Would also like to point out TMSU [0] which also presents a filesystem-y interface, alongside a quite powerful CLI. [0]: https://tmsu.org/ https://tmsu.org/

Fnoord 7y ago
Can it import from current file magic? ID3? CDDB? IMDB? You see, it is a big effort to manually input all this data. What is useful is reliable, public source entries. You also want a documented data format for future compatibility. So you can migrate to the next format. This furthermore gives the advantage that applications can use the metadata. Although I suppose the file system abstraction achieves the same?

amelius 7y ago

I think tags are a dead end because few users will make the effort to tag their files properly, at all times. Better use the Google approach, and let ML/NLP do the job.

eitland 7y ago
Users will not give their files proper names or put them into folders or tidy their desks or do the laundry either. That is some users won't do that, but by adapting everything to these users we are making everything worse for everyone including them.
- amelius 7y ago
  The problem is that everyone is one of these users at some point. E.g. when about to leave the office and receiving an attachment. Or at 3am when solving a tough problem. Instead of requiring every file to be properly tagged, why not take the Google-approach altogether?
  - eitland 7y ago
    Perfect example. However: I'm not trying to argue against search, I'm arguing against everyone having to rely on search because ux designers decided to remove every other way of finding files because someone might someday forget to tag the file or whatever the procedure is.