6 ms·

I think most people would agree a tag filesystem or a similar concept is a great idea. I have wanted one myself for a long time. Yet, for some reason, it doesn'

by yiyus 9y ago

I think most people would agree a tag filesystem or a similar concept is a great idea. I have wanted one myself for a long time. Yet, for some reason, it doesn't take off. Do you think it is just a problem of implementation?

Baeocystin 9y ago
Hierarchical file systems allow you to say for sure where a file isn't. Every tagging system I've ever played with has turned out to be a mess in actual use compared to the simplicity a hierarchy provides. (I am just a user, not a developer of filesystems, so this may be a naive opinion.)
- kijin 9y ago
  Tags are fun when you have a few thousand items to test your MVP with. It gets much less fun when you have millions of items with thousands of tags, all on a flat hierarchy. On the other hand, when you're stuck with a flat hierarchy anyway (e.g. thousands of pictures, all named DCIMxxxx.jpg), tags can be more useful. But only if they're automatic. I want the best of both worlds. I want to organize my stuff into folders and use tags to search for individual items. There's no need to be a purist on either side. "Designing better file organization around tags" is a good thing. "Designing better file organization around tags, not hierarchies" is not.
  - nayuki 9y ago
    I understand your concerns, and they are indeed valid. First off, I doubt that managing millions of files in a traditional hierarchical file system is fun either. You'd likely run into problems with making unique names, sharding folders, and categorizing files that logically belong in multiple places. I also have some worries that existing file systems (say NTFS or XFS) will behave or perform well with millions of files. I believe that implementing tags is a starting point for the problem of managing millions of files in a sensible way. Speaking of thousand of pictures, what I really want is to dump all my photos into one folder. Right now, tools are ill-equipped to deal with large folders, so I am forced to manually create a new folder for every thousand or so items.
    - cyphar 9y ago
      > I also have some worries that existing file systems (say NTFS or XFS) will behave or perform well with millions of files. I believe the mantra for XFS is "if you have large or lots, use XFS". XFS has a lot of optimisations for metadata operations which should mean it's better than most filesystems for lots-of-files and large-files cases (Dave Chinner has given several talks about the performance characteristics of XFS with "large or lots" cases).
  - enobrev 9y ago
    I absolutely agree with this, except to add that it seems, in theory, in the case of a 100% tag-based file-system, there would never (*very rarely) be a flat list where you have to scroll through millions of files. The UX of a single flat list with millions of files named DCIMxxxx.jpg is a limitation of the current format and doesn't make sense when so much information can be generated about our files upon creation. In this hypothetical FS, all files would have dynamically generated tags for created date/time, modified date/time, access date/time, originating application, given filename, owner, group, permissions, format, geo-data if available, originating hostname, EXIF data should all be first-class tags as prominent as the others, and so on. I visualize it to work, by design, something like how google photos UX works, grouping everything by EXIF data automatically before you ever start organizing things on your own. Just as well, with any files created manually, in the application "tagging" should be just as prominent a function as "naming" is right now.
    - TeMPOraL 9y ago
      Part of the adoption problem here would be trust. Hierarchical filesystems are something we're used to, and we can trust they're implemented correctly. That means, if I visit a folder and see some files, I know what I see is all of the files there (+/- hidden file settings); if something is missing, it's not there, period. Tag search is a search. Can be broken. Can be optimized in a way that causes it to lie. I look at the results, and I'm not sure if they're complete. Maybe the file I'm looking for is really not there, or maybe the search gave up too early. Or the tag was slightly misformatted? Maybe I'm too used to the old thing, but I like the notion that there's one canonical tree structure that makes all my data reachable. In case I've misplaced something, the search space of all paths through the filesystem tree (or a subtree of interest) is vastly smaller than the search space of all possible values of all relevant tags.
      - enobrev 9y ago
        You make excellent points, although I think I the issues you raise exist in our current filesystems as well. Provided the FS is indexed properly, opening a tag should show all files associated with that tag immediately. Just like opening (or listing) a directory does. And in the case of search, it absolutely sucks on today's filesystems. Don't get me wrong, find and grep are incredible tools. I simply mean that it's not like searching for files beyond the hierarchy is known for the pleasurable UX. The only way I know a grep of a whole drive or deep directory is done is because I get my blinking cursor back. At the very least with a proper tagging system, we would be inherently familiar with the indexes available to us.
lambda 9y ago
A few things I can see off the top of my head that would need to be solved for tag filesystems to be able to take off: 1. Inertia. Most software assumes hierarchical filesystems, and assumes it can control some portion of that hierarchy. This includes things ranging from search paths for various things ($PATH for binaries, library paths, etc), temporary files, preferences files for applications, assumptions made about hierarchical filesystems in archive formats like ZIP and TAR, etc. 2. Permissions. With a hierarchical filesystem, you can apply permissions on higher levels of the hierarchy to control access to lower levels, and you can have various forms of permission inheritance to control permissions on new files. Need a design for how to do that on tag filesystems. 3. Mounting. Filesystems come and go; some are on your OS drive, some are on removable media, some are network filesystems. Hierarchical filesystems means that each one has a single root, and it's easy to tell where the boundaries are. 4. Tagging taxonomy. What kinds of tags do you use? What happens if you mount a filesystem in which someone else used a different tagging taxonomy than you used? Who controls different parts of the tag space? What happens if you import an archive of material which uses a different tagging scheme than you use? 5. Projects. How do you group files of different types with different tags into discrete projects? How do you bundle related files together? How often would you want to see files by arbitrary tag lumped together, rather than looking in particular projects that have a pre-defined structure? 6. UI. How do you browse tags? How do you refine down? In many cases, rather than a general purpose tags based interface, you actually want media-specific browsers, like ones specialized for music which let you browse by artist, album, playlist, etc, or photo galleries that can show you previews of the photos, or video browsers which can show projects, bins, and sequences (for video editing), or IDEs which can either show you file hierarchy or allow you to browse by class, function, etc. 7. And finally, why is a tag-based filesystem necessary for this? What is wrong with the current approach, in which there are special purpose applications which can index, tag, and display certain types of media in certain ways? For instance, you can use your text editor or IDE to navigate among files within development projects, iTunes or Play Music or whatever to browse your music, Lightroom or Darktable or iPhoto or Google Photos to manage your pictures, iMovie or Final Cut or Premiere or Avid for browsing and managing your video, and so on. They all frequently have some way of tagging files, but also have specialized UIs for browsing the specific types of files they are defined for without having to do explicit tagging, and the actual files are just stored on a normal hierarchical filesystem. There are a couple of good thoughts in the original post, but a lot is handwaved away, such as mutability of files, which is an incredibly important use case, for a huge amount of what files are used for today. Lots of people have brought up the idea of making tag based or database based filesystems, such as the failed WinFS effort (https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS), but it's actually a pretty big problem to solve.
vbezhenar 9y ago
I'm not sure about that. I'm fine with hierarchical file system and links for my projects (desktop or explorer side bar). I tried few times to make use of macOS tags, but I didn't find them useful. I guess, for people who work with actual documents, it might be different. For me all those systems trying to be smart end up showing me billions of .class files from my build directories and wasting cycles trying to index them.

by yiyus 9y ago

Baeocystin 9y ago

Hierarchical file systems allow you to say for sure where a file isn't. Every tagging system I've ever played with has turned out to be a mess in actual use compared to the simplicity a hierarchy provides. (I am just a user, not a developer of filesystems, so this may be a naive opinion.)

kijin 9y ago
Tags are fun when you have a few thousand items to test your MVP with. It gets much less fun when you have millions of items with thousands of tags, all on a flat hierarchy. On the other hand, when you're stuck with a flat hierarchy anyway (e.g. thousands of pictures, all named DCIMxxxx.jpg), tags can be more useful. But only if they're automatic. I want the best of both worlds. I want to organize my stuff into folders and use tags to search for individual items. There's no need to be a purist on either side. "Designing better file organization around tags" is a good thing. "Designing better file organization around tags, not hierarchies" is not.
- nayuki 9y ago
  I understand your concerns, and they are indeed valid. First off, I doubt that managing millions of files in a traditional hierarchical file system is fun either. You'd likely run into problems with making unique names, sharding folders, and categorizing files that logically belong in multiple places. I also have some worries that existing file systems (say NTFS or XFS) will behave or perform well with millions of files. I believe that implementing tags is a starting point for the problem of managing millions of files in a sensible way. Speaking of thousand of pictures, what I really want is to dump all my photos into one folder. Right now, tools are ill-equipped to deal with large folders, so I am forced to manually create a new folder for every thousand or so items.
  - cyphar 9y ago
    > I also have some worries that existing file systems (say NTFS or XFS) will behave or perform well with millions of files. I believe the mantra for XFS is "if you have large or lots, use XFS". XFS has a lot of optimisations for metadata operations which should mean it's better than most filesystems for lots-of-files and large-files cases (Dave Chinner has given several talks about the performance characteristics of XFS with "large or lots" cases).
- enobrev 9y ago
  I absolutely agree with this, except to add that it seems, in theory, in the case of a 100% tag-based file-system, there would never (*very rarely) be a flat list where you have to scroll through millions of files. The UX of a single flat list with millions of files named DCIMxxxx.jpg is a limitation of the current format and doesn't make sense when so much information can be generated about our files upon creation. In this hypothetical FS, all files would have dynamically generated tags for created date/time, modified date/time, access date/time, originating application, given filename, owner, group, permissions, format, geo-data if available, originating hostname, EXIF data should all be first-class tags as prominent as the others, and so on. I visualize it to work, by design, something like how google photos UX works, grouping everything by EXIF data automatically before you ever start organizing things on your own. Just as well, with any files created manually, in the application "tagging" should be just as prominent a function as "naming" is right now.
  - TeMPOraL 9y ago
    Part of the adoption problem here would be trust. Hierarchical filesystems are something we're used to, and we can trust they're implemented correctly. That means, if I visit a folder and see some files, I know what I see is all of the files there (+/- hidden file settings); if something is missing, it's not there, period. Tag search is a search. Can be broken. Can be optimized in a way that causes it to lie. I look at the results, and I'm not sure if they're complete. Maybe the file I'm looking for is really not there, or maybe the search gave up too early. Or the tag was slightly misformatted? Maybe I'm too used to the old thing, but I like the notion that there's one canonical tree structure that makes all my data reachable. In case I've misplaced something, the search space of all paths through the filesystem tree (or a subtree of interest) is vastly smaller than the search space of all possible values of all relevant tags.
    - enobrev 9y ago
      You make excellent points, although I think I the issues you raise exist in our current filesystems as well. Provided the FS is indexed properly, opening a tag should show all files associated with that tag immediately. Just like opening (or listing) a directory does. And in the case of search, it absolutely sucks on today's filesystems. Don't get me wrong, find and grep are incredible tools. I simply mean that it's not like searching for files beyond the hierarchy is known for the pleasurable UX. The only way I know a grep of a whole drive or deep directory is done is because I get my blinking cursor back. At the very least with a proper tagging system, we would be inherently familiar with the indexes available to us.

lambda 9y ago

A few things I can see off the top of my head that would need to be solved for tag filesystems to be able to take off: 1. Inertia. Most software assumes hierarchical filesystems, and assumes it can control some portion of that hierarchy. This includes things ranging from search paths for various things ($PATH for binaries, library paths, etc), temporary files, preferences files for applications, assumptions made about hierarchical filesystems in archive formats like ZIP and TAR, etc. 2. Permissions. With a hierarchical filesystem, you can apply permissions on higher levels of the hierarchy to control access to lower levels, and you can have various forms of permission inheritance to control permissions on new files. Need a design for how to do that on tag filesystems. 3. Mounting. Filesystems come and go; some are on your OS drive, some are on removable media, some are network filesystems. Hierarchical filesystems means that each one has a single root, and it's easy to tell where the boundaries are. 4. Tagging taxonomy. What kinds of tags do you use? What happens if you mount a filesystem in which someone else used a different tagging taxonomy than you used? Who controls different parts of the tag space? What happens if you import an archive of material which uses a different tagging scheme than you use? 5. Projects. How do you group files of different types with different tags into discrete projects? How do you bundle related files together? How often would you want to see files by arbitrary tag lumped together, rather than looking in particular projects that have a pre-defined structure? 6. UI. How do you browse tags? How do you refine down? In many cases, rather than a general purpose tags based interface, you actually want media-specific browsers, like ones specialized for music which let you browse by artist, album, playlist, etc, or photo galleries that can show you previews of the photos, or video browsers which can show projects, bins, and sequences (for video editing), or IDEs which can either show you file hierarchy or allow you to browse by class, function, etc. 7. And finally, why is a tag-based filesystem necessary for this? What is wrong with the current approach, in which there are special purpose applications which can index, tag, and display certain types of media in certain ways? For instance, you can use your text editor or IDE to navigate among files within development projects, iTunes or Play Music or whatever to browse your music, Lightroom or Darktable or iPhoto or Google Photos to manage your pictures, iMovie or Final Cut or Premiere or Avid for browsing and managing your video, and so on. They all frequently have some way of tagging files, but also have specialized UIs for browsing the specific types of files they are defined for without having to do explicit tagging, and the actual files are just stored on a normal hierarchical filesystem. There are a couple of good thoughts in the original post, but a lot is handwaved away, such as mutability of files, which is an incredibly important use case, for a huge amount of what files are used for today. Lots of people have brought up the idea of making tag based or database based filesystems, such as the failed WinFS effort (https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS), but it's actually a pretty big problem to solve.

vbezhenar 9y ago

I'm not sure about that. I'm fine with hierarchical file system and links for my projects (desktop or explorer side bar). I tried few times to make use of macOS tags, but I didn't find them useful. I guess, for people who work with actual documents, it might be different. For me all those systems trying to be smart end up showing me billions of .class files from my build directories and wasting cycles trying to index them.