22 ms·

Designing better file organization around tags, not hierarchies (2017)

• rasengan0 5y ago
  I ditched everything for a simple Zettelkasten in PlainVanillaVim all on a SdCard EverythingIsAFile NoExtensions CamelCase hit gf on a word/file/tag and enjoy freedom :map gf :e <cfile><CR> https://vimhelp.org/editing.txt.html#gf https://vimhelp.org/editing.txt.html#gf sprinkle in RipGrep Fzf CtrlP as needed
  • megous 5y ago
    This is tempting, at least for note taking. :)
  • klodolph 5y ago
    I like to organize files in a similar way but use spaces in filenames. I find it a lot easier to read filenames that way, and in the shell, <tab> will add all the necessary quotes / backslashes.
• dsr_ 5y ago
  The difference between tags and hierarchies is this: if you don't know what you're looking for, hierarchy can guide you.
  • arduinomancer 5y ago
    Couldn’t you just look at a list of tags?
    • dsr_ 5y ago
      Sure. How many tags are in your system? The number of tags should be more or less similar to the number of non-leaf nodes in a hierarchy, or else you aren't capturing the same information. Any tag that applies to more than half of the files is probably useless. On my blog, the tags "blog" and "technology" are definitely useless. That's fewer than 250 entries and already it has cruft. Were people consistent when they added tags? Does your system suggest tags automatically? Is this actually a full-text search minus stop words? Is there a librarian who cleans up after you and merges tags that have the same meanings? Would the full-text search be more useful than tags?
    • dredmorbius 5y ago
      I have this problem in Pocket. I use ... a lot of tags. (I couldn't tell you without downloading my entire archive and doing some magick to count them, as Pocket won't give me a count itself.) Pocket lags by well over a minute in even starting to populate my tags list after I enter some text into the field. (Chrome/Android, using an external keyboard.) It similarly takes several minutes to scroll through the tag list using either the Web client or the Android app. You could argue that this is a piss-poor UI/UX issue, and I'd agree. It's piss-poor UI/UX issue that Pocket have failed to address since I first reported it over five years ago. https://old.reddit.com/r/dredmorbius/comments/5x2sfx/pocket_it_gets_worse_the_more_you_use_it/ https://old.reddit.com/r/dredmorbius/comments/5x2sfx/pocket_... If you'd like another list that has rapid-scannability issues, there's the Library of Congress Subject Headings: These are publicly available. In 27 PDF files, corresponding to printed books ranging from 12 to 842 pages in length: https://www.loc.gov/aba/publications/FreeLCSH/freelcsh.html https://www.loc.gov/aba/publications/FreeLCSH/freelcsh.html
  • Rygian 5y ago
    That's just bad UX of the tag browsers.
    • dredmorbius 5y ago
      If that's the case, how specifically would you improve that UX?
      • wander_homer 5y ago
        Have the option to view the tagged items in a tree, which is dynamically created from the tags. For example my audio player does just that, e.g. `%artist%/(%year%) %album%/%tracknumber%.%title%`
        • theon144 5y ago
          Also known as "facets". Sometimes you might want to browse "year/album/artist" too (think Various Artists / compilation albums), instead of the more common "artist/year/album". Or even "artist/year/album" for the more prolific artists out there. Progressively "drilling down" is super low-hanging fruit with regards to tag-based organization, it's seriously neglected.
          • dredmorbius 5y ago
            Thanks, that's ... closer to what I have in mind. I also agree that the lack of drill-down on tag-based organisation ... kills about 99.9996% of the value. It's sufficiently valuable that you'd expect this to be part of basic implementation libraries by now.
        • dredmorbius 5y ago
          How are relationships between tags determined or specified? A tagging system is flat by default. Though that makes rendering the tree somewhat easier...
          • wander_homer 5y ago
  • dredmorbius 5y ago
    Another key difference is that a hierarchical (or positional) attribute is acquired inherently by a document's position within that hierarchy or location. In both cases, the hierarchy is inherently positional, affords locality, and guides both the reader and the document by asserting that locality inherently. If you're in a place, then that place gives nearness or farness from objects. If you're an object, you're in some location inherently. Tags, in this sense, are placeless, and don't provide attributes of or guidance to nearness. At least not without some additional mechanism.
• jameskeim 5y ago
  Didn't the BeOS filesystem (BFS and later OpenBFS) make this possible? I remember being impressed with they way extended attributes made the file system and files into a database for any programs that wanted to interact with them. And, I don't know exactly how it worked but the system supported live queries that allowed you to drill down based on the attributes. MacOS and Windows both created things intended to appear like live queries but they always seemed slower and less elegant.
  • kitd 5y ago
    OS/2 allowed custom file attributes too, with search & retrieval.
  • waddlesplash 5y ago
    Yes. Attributes: https://www.haiku-os.org/docs/userguide/en/attributes.html https://www.haiku-os.org/docs/userguide/en/attributes.html Queries: https://www.haiku-os.org/docs/userguide/en/queries.html https://www.haiku-os.org/docs/userguide/en/queries.html
• dusted 5y ago
  Reminds me of an old project I did where I built a fuse system for this, files still lived on a normal ext3 fs, but the overlay presented files as tag paths, yo you could access, for instance, a movie like /tagfs/movies/year/1999/matrix.avi or /tagfs/movies/genre/scifi/matrix.avi There was a special path /tagfs/untagged/ which listed any files that didn't have at least one tag.
• fungiblecog 5y ago
  tags are great when you know what you're looking for but are terrible for browsing a new dataset. this is especially true when you're the newbie eg on a project and you didn't have any input into the tag structure.
  • japanuspus 5y ago
    This. Hierarchy allows a simple manual traversal, leading to better discoverability.
    • sgc 5y ago
      Any tag based system should have a breadcrumb function that allows for manual traversal.
    • kmort 5y ago
      I've recently built a tagging system based on SKOS [1]. This supports hierarchical as well as associative relationships between tags (while not strictly requiring either), as well as ad hoc groups of tags. While SKOS was intended for more formal vocabularies, I've found its use as a basis for a tagging system makes exploration and navigation of a topic area reasonably organic, as it allows users to specify relationships only as they see as fit and intuitive. [1]: https://www.w3.org/TR/skos-primer/ https://www.w3.org/TR/skos-primer/
  • sgc 5y ago
    There just needs to be a default view that shows you the most important tags to start from - either by most used files, or largest number of files, or most active recent changes, or as managed by someone. The nice thing about tags is all of those could be top level tabs and you are sailing.
• tiberiusteng 5y ago
  If using Windows, use Everything (https://www.voidtools.com/ https://www.voidtools.com/) and when naming a new file treat the filename as a tag set. Then Everything could become your shell ...
• Razengan 5y ago
  One of the best uses of tags is to let a file effectively exist in multiple “folders” For example I have folders of screenshots named after various shows and games, and I use tags to further organize the images based on their suitability for different “reactions” on online forums :) So naturally I have hundreds of tags but macOS doesn’t seem to keep up with that many and after a certain point it feels like Apple have forgotten about tags and improving their integration into the system.
• pininja 5y ago
  I like how this article is laid out as first defining the existing systems used today since while it’s all I know, I haven’t spent the time defining it. And then describes a number of inspirational examples of how it could be different. Desktop file systems seem impossibly hard to change at this point, but cloud storage and mobile file systems are still so new and not amazing in my opinion - there’s still hope for a better experience.
• spicybright 5y ago
  I've been hearing this for years, but I've never had the mental model for it. Maybe I just have to try it out. There's a very neat project (can't remember the name, and some details below may be wrong,) that mounts a FUSE filesystem that uses only tags. Paths have identical syntax to hierarchical ones, only each segment of a path is a tag. so `/document/taxes/2020` would return a collection of files, but you could also write `/taxes/2020/document` and it would mean the same thing. The advantage is you can use standard unix tools like ls, mv, cp, etc. as well as R/W using standard programs (like how you'd export an image.) I'm sure there's edge cases to it, but I always found the idea very neat. Anyone remember what I'm talking about?
  • im3w1l 5y ago
    This might work for documents but a difficulty I see is how you would handle system files. See, with documents it's fine to always return a collection of files, but for a system file say /etc/passwd, you want to make sure you get exactly (status quo guarantees at most which is almost but not quite as good) one file.
    • jbergens 5y ago
      Just think of the endless possibilities of a node_modules folder replaced with some tags! /s
  • oneplane 5y ago
    While I don't remember that specific project's name, I do remember that a lot of that was possible because files in a filesystem often aren't just some hierarchical folder structure. There is often a tree-like structure to be able to find a single file quickly amongst many files, but there was no technical reason why it wouldn't also support off-tree indexing based on tags. I imagine one of the problems you'd run in to is mass updates since deleting a directory with many files in it below a single tree node is very much limited to that part of the filesystem's tree, but doing it with tags causes updates all over the place. It does remind me of this more metadata-like FS: https://github.com/marook/tagfs https://github.com/marook/tagfs As well as the thing Apple does in Finder where it has a tag database and dynamic 'tag' directories where it shows you everything that was tagged with that tag. Then the search function would allow selecting files based on tags so you can do many-tag searches and only find the files that match all of them. I think that one is based on an in-filesystem metadata stream.
• polote 5y ago
  (2017)
  • dang 5y ago
    Updated. Thanks!
• tdrdt 5y ago
  The same is happening in webshops. Products can be assigned to hierarchies (categories) but in shops like Amazon it is obvious it is more like tagging.
  • indymike 5y ago
    In ecommerce, products are often presented to the shopper by search facets, so kind of all of the above applies.
  • kzrdude 5y ago
    And amazon thanks to that is opaque, I never look around for stuff, just search and find or not. They could have made something better where you can discover other stuff.
• Tarsul 5y ago
  ok, maybe this is the right thread for this question: Anyone has a good structure for their own mp3s? I'd imagine tagging all files and using those tags for playlists would be a good way to do it. Does anyone do something like this or similar and can give pointers?
  • crooked-v 5y ago
    Run an old version of iTunes in a VM? The column browser is still one of the generally best media browsing interfaces I've ever dealt with.
  • sparkie 5y ago
    Use MusicBrainz Picard. The MusicBrainz database is probably the best there is, and it does a good job of automatically tagging your music. Once tagged, every file will have a unique musicbrainz_trackid (UUID) in its metadata, which can be used to recover/update the metadata associated with the track automatically from the database, which is constantly updated (and to which you can contribute if it is missing metadata for your tracks). You can configure Picard to arrange files and rename them however you want. It has some simple scripting functionality so you can name things conditionally based on the presence or absence of metadata, etc. [https://picard-docs.musicbrainz.org/en/tutorials/naming_script.html https://picard-docs.musicbrainz.org/en/tutorials/naming_scri...] If you are concerned about privacy, you can run your own musicbrainz instance in a VM and download a copy of the entire database. Picard is extensible with Python. There's some existing plugins for generating playlist files.
    • ghostly_s 5y ago
      A complementary alternative I'd suggest is beets[1], a front-end agnostic CLI tagging utility that also matches your files against the MusicBrainz database and can both correct the ID3 tags and maintain a directory hierarchy based on those tags. The biggest shortcoming of the MusicBrainz database I've found so far, however, is genre tags. Most releases seem to have only one or a handful of genres listed with no consistent genre hierarchy convention, but I've been experimenting with an extensions that pulls genre tags from discogs. 1. https://beets.readthedocs.io/ https://beets.readthedocs.io/
• throwaway984393 5y ago
  I think the author is just a little bit behind the times in terms of organizing large corpuses of inter-related data. Tagging as a general idea is too fast-and-loose without a system/structure to organize the tags. Semantic Web stacks are one way to improve on this using taxonomies and ontologies, query languages and data specs; they almost mention it in the alternative systems ("touples") but don't dig into just how hard it is to manage data using unstructured references (or even structured ones!). Simple hierarchies are ..... simple. Tags are simple too, but they quickly devolve into new complexities as people try to figure out how to apply them, find them, organize them. Hierarchies aren't typically as difficult to manage because it boxes you into re-creating the same mental model for organization, just with different classifiers for each level of the hierarchy. They're less flexible, but they're easier to grok, maintain, and use. My major concern is that there isn't really a need to "fix" hierarchies, it's just a nagging problem that someone doesn't want to deal with, so their solution is to make something more complicated.... and more complicated might not make it better. It should also be feasible to design applications to organize the files without having to rewrite filesystems.
  • pimlottc 5y ago
    You seem to know a lot about this; what is considered the "correct" way to organize data like this these days?
    • throwaway984393 5y ago
      I couldn't claim to know what is "correct" and what isn't! I've just worked on projects to organize large collections of interrelated datasets (for example, to update correlations between concepts, to make search engines more effective, to identify related or dependent item relationships, etc) and for our project we used a Semantic Web stack. Browsing GitHub for "knowledge graph" or "knowledge management" seems to pop up some cool looking projects, but I think everyone is still trying to figure out what works for a particular use case rather than generally. I hope a real data scientist can reply with whatever the latest and greatest solutions are. Semantic Web tech for knowledge graphs are continuing to evolve, but also kind of old, and they're still mostly used for research projects. Part of that is probably because the terminology is unusual, and implementation leads you down a long rabbit hole of new and confusing concepts. So that's why I'm thinking that just sticking to a boring inefficient hierarchy might not be so bad...
• jasode 5y ago
  The previous 2018 discussion had comments from the author: https://news.ycombinator.com/item?id=16763235 https://news.ycombinator.com/item?id=16763235
• deleted 5y ago
  [deleted]
• webwielder2 5y ago
  Tags vs. Hierarchy are like the WFH vs. Office debate: something people assume has a right answer but is actually personal preference.
  • jbergens 5y ago
    I don't think it is just a preference for system files and application files. For documents, sure. Think about an application that tries to find the settings file. Which tags should it use in the search? How does the user know that no other file on the computer will suddenly be found because they used the same set of tags?
    • lucasyvas 5y ago
      It would not happen - uniqueness would be enforced by the file name and its set of tags. Only one match could exist.
• encyclic 5y ago
  Wasn't this the intent behind Windows Vista - a tag-based DB-as-filesystem with hierarchical paths just one "lens" through which to view the DB? I use Google Drive this way, largely through search rather than directory-based organization, though I do also employ that for often-used collections.
  • sigg3 5y ago
    It's the entire point of MS SharePoint afaict: create a tag based FS on top of some SQL database.
    • mousepilot 5y ago
      ok well if its the entire point of sharepoint I can conclude that tags suck and go read something else because, well, sharepoint.
  • ximeng 5y ago
    https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS WinFS - was cancelled
• btrettel 5y ago
  From the article: > ### Hard and soft links as non-solutions > Soft links have a nearly opposite set of problems as hard links – soft links can span different file systems, but they generally don’t track the target files getting moved or renamed (except that Windows provides such a system service, but may not be reliable); while hard links are all indistinguishable, some application software behaves differently on soft links than on real hard-linked files. In spite of these problems, hard and soft links still require an exponential amount of effort to classify a set of files in multiple ways, and require the user to manually remember all the possible paths that a file can be reached from (important when editing and removing files, not important when browsing/retrieving files). They are non-scalable kludges compared to true tagging. As a fan of hierarchies for file organization, and as someone with quite literally thousands of soft links, when I read things like this, I don't know if the person arguing against links has tried this approach. It works totally fine for me. Yes, with soft links, moving the original file breaks the links. I wrote a fairly simple bash script to automatically fix these for my reference PDF files. It works because each PDF file I save has a unique file name. So figuring out where the links need to point is pretty simple. That makes me a "power user", I guess, but the author is at least at the same level and I think could figure it out. With respect to the "exponential amount of effort to classify a set of files in multiple ways", I guess the author is referring to navigating the hierarchy to link a file in multiple places? I use tagging at my work, and I personally find scrolling through my list of about 200 tags to be comparable in terms of time to navigating through a hierarchy. The bottleneck is the human. Edit: Here's another article by someone who is not a fan of links: https://karl-voit.at/2018/08/25/links/ https://karl-voit.at/2018/08/25/links/
  • mro_name 5y ago
    immediately thought of Karl Voit as well. https://karl-voit.at/managing-digital-photographs/ https://karl-voit.at/managing-digital-photographs/ and his talk at the Grazer Linuxtage 2018.
  • livrem 5y ago
    I use hard and soft links (mostly the former for files and the latter for directories) and I think it works very well in practice despite the theoretical problems brought up.
• civilized 5y ago
  Tags are superfluous if you have a good search engine. Hierarchy isn't.
  • guerrilla 5y ago
    Yeah, tagging really seems like the job of a file manager (and indexer) rather than a file system. That seems like an easy way to get everything we want without rewriting billions of lines of code that deal with hierarchies.
  • dredmorbius 5y ago
    Tags (or other metadata) are: - A controlled vocabulary. This addresses the problem of numerous terms referring to the same concept, the same terms applying to different concepts, disagreements on spelling or charactersets, and standardisation or cross-references between multiple terms. - Externally applied, subject to authority independent of an author or publisher. This addresses the problem of keyword-stuffing. - Capable of referencing information not within a document itself. Its place of publication, earlier or later versions, cultural context, citations, amongst others. - Can be applied to nontextual media: images, audio, video, data. Tags themselves are not hierarchical. Search is not introspective or contextual.
    • Tomte 5y ago
      Tags could be a controlled vocabulary, but in almost all cases they are free-for-all. Archive of Our Own has an interesting scheme, where authors tag their pieces freely, and volunteers behind the scenes enrich those pieces with tags that are indeed from a controlled vocabulary. (They also do other mind-boggling stuff: in those tags they encode all kinds of information, so that you can search for a Kirk-Spock love story where violence is involved and Spock is dominant, and so on…)
      • dredmorbius 5y ago
        I could have been clearer, but tags, even in a highly informal process, *are supplied by the reader(s) or curator(s) rather than the author/publisher, at least in the context of organising one's own archive. In that sense, tags are, even if not highly structured, a controlled vocabulary. (I'm not referring to tagging that's provided by a publishing site itself, though yes, that's a fairly common practice. Tagging by a skilled third-party curator or librarian can of course be excellent.)
• austincheney 5y ago
  For tags I use meta data, which are included as mapped text at the extreme end of many media formats. For example ID3 data on MP3 files. I have found this incredibly helpful for MP3s because I have thousands of them and there are many similar names. Windows Explorer provides columns for this data in its detailed file system view (not by default) which is incredibly helpful and trivial to customize. For everything else folders are enough. I have hundreds of movies on a hard disk and yet folders are enough. When I do need more the data I want is generated by the file system: last modified, file size, and so forth. What improves file usability the most for me is network access by meta data. For example Windows Explorer and OSX Finder are nice but I would rather have the exact same interface on the same local machine for a bunch of remote machines regardless of their file system or operating system. Then copy to a different machine is just drag and drop from one window onto another in an application that looks like some local OS, that windowing interface needs to allow sorting and filtering and search by meta data just like Windows Explorer. Having an application that does this for me has been great.
• xixixao 5y ago
  I love Bear with it’s 1+ hiearchical tags per note. It’s the perfect combination of both worlds for me. Would encourage the author to check it out (if it’s any different).
• fmajid 5y ago
  Librarians have been classifying the world's knowledge since forever, and they developed faceted classification systems (Ranganathan, 1933) to deal with these issues. https://www.researchgate.net/publication/321840994_Ranganathan_and_the_faceted_classification_theory https://www.researchgate.net/publication/321840994_Ranganath... A notable one that hierarchies embed the point of view of the classifier, e.g. the Dewy Decimal's ridiculous classifications of religions (codes 200–299), making minute distinctions like 285 (Presbyterian, Reformed, Congregational), 286 (Baptist, Disciples of Christ, Adventist), then putting all non-Christian religions under a handful of afterthought headings 292–299: 294 for Hinduism, Buddhism, Sikkhism and other religions of Indian origin, 295 for Zoroastrianism and its descendants, 296 for Judaism, 297 for Islam and Bahaism lumped together, and 299 for New Age. Unfortunately, most of us do not have access to the services of a librarian to develop a taxonomy that corresponds to our own point of view then classify our files accordingly, which is why simple hierarchical taxonomies have endured and faceted ones seldom beyond specialized applications like Digital Asset Management.
  • sethammons 5y ago
    https://johnnydecimal.com/ https://johnnydecimal.com/ After discovering this on HN, I've wanted to put it into action. Seems like a streamlined and simple way to mimic Dewey.
  • maratc 5y ago
    Dewey Decimal System has only one (often-misunderstood) task: organizing bookshelf space. It does not have a task of classifying world information. The reason that fairly narrow "Presbyterian, Reformed, Congregational" topic has one DDS code and an extremely wide "Hinduism, Buddhism, Sikkhism and other religions of Indian origin" topic has one DDS code is simple: an average library has similar book-width for both topics.
    • tobr 5y ago
      An average American library, you mean.
      • 5y ago
• gavinhoward 5y ago
  Good ideas, but there are a few things wrong with this. First, we forget that filesystems are not hierarchies, they are graphs, whether DAG's or not. [1] Second, and this follows from the first, both tags and hierarchy are possible with filesystems as they currently are. Here's how you do it: 1. Organize your files in the hierarchy you want them in. 2. Create a directory in a well-known place called `tags/` or whatever you want. 3. For every tag `<name>`, create a directory `tags/<name>/` 4. Hard-link all files you want to tag under each tag directory that apply. 5. For extra credit, create a soft link pointing to the same file, but with a well-known name. This allows you to use the standard filesystem tools to get all files under a specific tag. For example, find tags/<name> -type f (The find on my machine does not follow symbolic links and does not print them if you use the above command.) If you want to find where the file is actually under the hierarchy, use find -L tags/ -xtype l Having both hard and soft links means that 1) you cannot lose the actual file if it's moved in the hierarchy (the hard link will always refer to it), and 2) you can either find the file in the hierarchy from the tag or you know that the file has been moved in the hierarchy. Of course, I'm no filesystem expert, so I probably got a few things wrong. I welcome smarter people to tell me how I am wrong. [1]: https://lobste.rs/s/ydno8w/tree_structure_file_systems#c_njgkbj https://lobste.rs/s/ydno8w/tree_structure_file_systems#c_njg...
  • jeddy3 5y ago
    > 2. Create a directory in a well-known place called `tags/` or whatever you want. > 3. For every tag `<name>`, create a directory `tags/<name>/` > 4. Hard-link all files you want to tag under each tag directory that apply. Does this only give you one level of tags? (i.e you can't combine tags when exploring)
    • ghoward 5y ago
      It does, unfortunately. But you probably could implement finding something with two tags with some command-line fu. My first crack at it is this: find -L tags/tag1 tags/tag2 -xtype l | sort | uniq -d I don't know if that would work (on mobile; can't test), but from reading the man pages, it seems like it would do the trick.
    • 5y ago
• Kalanos 5y ago
  Tags are the lazy man's schema
  • dredmorbius 5y ago
    Schemas are an ignorant / arrogant person's tagging system ;-)
• lmilcin 5y ago
  Or better idea. Use directed graphs. Directed graphs are strictly a superset of both tree and tag (set) functionality. Tags (sets) can be thought as a special case of directed graph. You make the tag a node in the graph and you make files also nodes and have edge pointing in direction of the tag node. You can then do graph queries to find files tagged in a certain way. But graphs offers so much more. Because they are superset of both trees and sets, you can use it to represent both, at the same time. It is not very useful to have a lot of things tagged the same way because you end up with just a long list of things. Whereas in a graph you could say that you want to find objects from which you can reach certain tag node and all these objects can still have their own structure and even be part of multiple structures. For many years I had this idea to build my PIM where I could make arbitrary nodes being anything that could let me connect anything to anything. A node could be an email, a file, a link to external external website, a task, a contact, a reminder, etc. And you could connect everything to anything and have, for example a project that has important emails attached to it, the email could have attached a reminder to respond and a file that you want to include in response, and a note. You could browse this graph as a tree because locally it could be interpreted as a tree, you open the tree one level by finding all elements that pointing to the node.
  • MrLeap 5y ago
    How do you transform a digraph with cycles into a tree without? Seems perilous.
    • deleted 5y ago
      [deleted]
    • lmilcin 5y ago
      How do you traverse a symlink that points to one of its parent folders? How do you traverse a web of interlinked sites? Somehow Linux, Linux software, web browsers etc. are already able to deal with directed graphs.
      • MrLeap 5y ago
        You can handwave cycles created by symlinks away by ignoring them. Full traversal is possible relying only on the tree. If all you have is the digraph you have to rely on cycle detection. Transforming it into a hierarchy of depth requires snipping edges somewhere. I'm just gently disputing the claim that a digraph can be mapped 1:1 to a hierarchical filesystem. I'm open to being wrong.
• nathanmcrae 5y ago
  I find that using Everything search (https://www.voidtools.com/ https://www.voidtools.com/) makes me use the filesystem more like a tagging system. I still name files and directories meaningfully, but I don't worry about the hierarchy at all. Then when I want to get something, I just search (parts of) the terms I want and see the matching paths instantly. Using the filesystem hierarchically now feels painfully slow and awkward.
  • _dain_ 5y ago
    I do this with fd and ripgrep, which also lets me do full-text search through the files themselves. My filesystem has become much flatter and coarser-grained; just a few very large categories.
• jcelerier 5y ago
  I remember trying to use tags to tag my media collection a decade ago and miserably failing
• s1k3s 5y ago
  Just cancelled my Google Drive subscriptions last week due to this, and moved to SFTP on rented metal. If you can’t 100% predict what I’m looking for then don’t even bother. Just give me files and folders.
• megous 5y ago
  Usefullness of this is highly content specific. It maybe works for mp3s or videos/photos you made yourself and contain some metadata. I can't imagine tag based organization of all the random 2 mil. files I have that don't fit into these neat categories. I don't need access to most of these files unless I'm working on something relevant to them. When I work on X, I go to directory X and everything I need sits below X in some hierarchy. I'm never interested in anything that's below X unless I'm working on X. I don't want stuff from X to pollute some global namespace just because some mp3 or PDFs are present under X. That's true for hundreds of personal projects and tools I've made. Directory hierarchy is a pretty neat abstraction to me and all the tools I use already support it well.
  • Rygian 5y ago
    Does it mean that you will store multiple copies of a file, if it ends up being useful for multiple projects? About the random 2 mil. files you have lying around, the tags you should set should describe not the files themselves, but the reason why you chose to keep those files.
    • pgcj_poster 5y ago
      I have a similar system to GP, and the answer is yes: I will duplicate files that are relevant to multiple projects, with the exception of research papers, which all live together in Zotero.
    • megous 5y ago
      Yes. Pretty much. The reason why I'm chosing to store the files is the project they're part of. I don't need them otherwise. If the project is inactive or dead, I'll compress the folder into some archive and delete the folder. If something is meant to be useful for multiple projects it either gets symlinked, or put into its own project. Anyway, I'm not saying tagging as a primary categorization interface is not useful at all. It's just that it has limited applications where it would be definitely more useful than traditional filesystem hierarchy.
  • geysersam 5y ago
    A tagging system seems to me to be a superset of the hierarchical model. Just tag the files '/projects/a/src/module/name/' etc. and perhaps another tag for file type, and the models are more or less the same.
• slaymaker1907 5y ago
  I don't like the idea of file hashes for storing files. That really only works well for a limited set of files completely ignoring things like databases, note files, etc. If you want to generate a name, UUIDs would make way more sense. Additionally, even for files which never change, what happens when a bit gets flipped inside of the file? That would presumably change the hash without knowledge of the file system until you try and read it. For drives intended for consumers, you really don't want to have more data loss than is absolutely necessary in case of minor corruption. On a different note, I use Tiddlywiki with tags a lot for my personal notes and I think a hybrid approach with tags and a hierarchy works best. Hierarchies are useful for generating meaningful unique names where it is nice to have the notes be meaningful. Another thing that would be useful with a hybrid system is to have rules like /Skyrim/Guilds/ThievesGuild/NPC/Mercer automatically assign tags to Mercer by virtue of his placement like {"Skyrim", "NPC", "ThievesGuild"}. You could do this automatically via the path or by some sort of rules engine. Finally, I would really like it for a more complex information storage system to fully support 1, 2, and 3-tuple metadata. A 1-tuple is a tag, a 2-tuple is a key-value pair associated with a file, and a 3-tuple would be a relation between files with an optional key/value (some relations would be merely a tag while others might want a key/value). 2-tuples are obviously useful since they exist in limited form currently via file attributes, though they unfortunately are not exposed very well to users even though some file systems support arbitrary key/value pairs as attributes. 3-tuples are a little more niche, but I think it would be useful for keeping track of stuff like what file imports another. Humans probably wouldn't generate the metadata in the previous example, but it would help tools play nicely together since you could have a small static analysis tool which simply updates this metadata which could then be used by other tools or by users directly. One of the greatest things about file systems is that they allow for separate tools to interact with the same data in a structured way.
  • geysersam 5y ago
    Bit flip issue could probably be solved using error correction codes in the implementation of the file system. Interesting idea using 3-tuples for storing relationships between files.
• hyperpallium2 5y ago
  MS was going to use a relational filesystem at one time.
• agumonkey 5y ago
  I had this idea long ago but at the same time I worry about things too fluid. Both for performance and both for information efficiency. Tree can be seen as a preemptive good enough tag order. Some obvious dimensions like category, time will always be of use.
• vinodkd 5y ago
  Some really interesting ideas in a well organized article. Bookmarked!
• techsin101 5y ago
  it's easier for me to put folder "summer photos 2012" under photos than tag everything. In some cases it'd just bring garbage, i.e. script tag -> get ready of tsunami of script.js that got downloaded with html files i downloaded. I'm saying I can't think of a scenario where it'd be mostly helpful but that don't know whether there are other scenarios where there are such scenarios where tags are better.
• marginalia_nu 5y ago
  I've been doing a hybrid approach for a hypertext system I built. All documents have a canonical position in a filesystem, but are arbitrarily "taggable". Tags themselves are also files, but the property of "having a tag" is encoded as one file linking to another. The only difference between a tag and say an article is that the tags are in a different directory (not that they need to be, but it makes it easier to parse as a human). All documents automatically list internal backreferences, which means the tag files become useful for navigation.
• gorgoiler 5y ago
  I always see tags as a patch for when search doesn’t work effectively. If I have a hold of some content that I will want to find later and it doesn’t have the metadata needed to show up in a future search, I have to add that metadata myself. My solution is that all content goes into a new folder. Projects get meaningful folder names so that at least a search for “School” will find “School Trip 1997”, even if the documents in that folder “PASSPOR~.PDF” don’t themselves show up in the search results. In my experience it’s been rare where tags have been used to put data in more than one place.
• ggm 5y ago
  Specific to photos, and most commonly scanned photos is the problem of "approximate" or "uncertain" dates. The various blogs (exiftool forum &c) all discuss different ways of coercing "probably 1952" or "before 1940 and after 1920" into the EXIF structure. I'd love to hear of ways to leverage tags to do this. perhaps assign a specific date (the blogs often say 1952-01-01:01:02:03 as a datetime, to signal "uncertainty") could be improved on by an "uncertain" tag. There's also the syntax of tag separation inside EXIF. The various systems can't entirely agree what notation to use.
  • ogurechny 5y ago
    > 1952-01-01:01:02:03 as a datetime, to signal "uncertainty" I instantly thought that least significant fields are used to indicate uncertainty range (10 days around 1952-01-01, or 10 years around 1952?). After all, the older the photo, the less sure we can be about its timestamp (usually), so we can use even/odd seconds as a flag to indicate exact time or uncertain period length (log scale). It would be quite an evil hack of a common fixed timestamp format.
    • ggm 5y ago
      That's very clever. I thought about using alternate daytime fields to signal the degree of uncertainty as a magnitude to the declared date Time, which then becomes the centre. Or you can use some positive/negative signals to show not before or not after.
  • mceachen 5y ago
    ExifTool handles zeroes in datetime strings (I had to add that support to PhotoStructure a couple months ago!), but I suspect most other applications will croak on those values. I'd also really like a way to identify very-rough dating. A (custom?) tag could encode "precision", maybe with a time units, so a +/- 10 year "precision" would handle your "between 1920 and 1940" case. If you've got a suggestion, I'm interested: PhotoStructure has a ton of users with this same issue.
    • ggm 5y ago
• oftenwrong 5y ago
  Here's a relevant, "classic" article along the same lines: Clay Shirky - Ontology is Overrated: Categories, Links, and Tags (2005) https://web.archive.org/web/20191117162526/http://shirky.com/writings/ontology_overrated.html https://web.archive.org/web/20191117162526/http://shirky.com...
  • joshu 5y ago
    He spelled my name wrong and now it's preserved for all posterity. Argh
• karmakaze 5y ago
  While we're at it, go beyond overwriting a single content stream per name. VMS had ;version numbered suffixes for versions of each file. Git follows a model where each unique content item is a blob and there are references to it from different names and versions. Lots of 'filesystems' didn't have directories, or did have them but were implemented in a flat structure: mainframe OSes and LAN filesystems like Novell, which I discovered one time when an entire drive got all it's files deleted by some bug. It was possible to undelete them from the LAN 'trash' where all the dir/file names were in a very long flat list.
• novok 5y ago
  Hmmm, I wonder why they don't talk about the notion / wiki style of data organizational model with links inside this article. I guess because it's more text document format dependent.
• walterbell 5y ago
  Wildland (from architect of QubesOS, blockchain bits are optional) is tackling this problem: bottom-up ontology graphs that can be meshed into a global address space, items can appear at multiple places (similar to Bear) in the hierarchy. https://wildland.io https://wildland.io whitepaper: https://golem.foundation/resources/documents/wildland-w2h.pdf https://golem.foundation/resources/documents/wildland-w2h.pd... discussion: https://news.ycombinator.com/item?id=27473180 https://news.ycombinator.com/item?id=27473180 videos: https://vimeo.com/user124198022 https://vimeo.com/user124198022
• asielen 5y ago
  Hierarchies vs tags has come up for me in my recent effort to scan and organize thousands of old family photos (100+ years old). My main goal is to make the data as future proof as possible, in my mind that means going with some standard filesystem hierarchies but supplementing it with photo metadata using exiftool. I don't trust any tagging solution to be useable in 5+ years that isn't based on EXIF/XMP standards and that only really works for photos (and not all formats). While you can force write XMP standard metadata onto unsupported file types--there is a push to standardize XMP across all files--metadata support is far from standard across operating systems or tools. For my purposes I have a folders with Ahnentafel numbers so they can be quickly found: https://en.wikipedia.org/wiki/Ahnentafel https://en.wikipedia.org/wiki/Ahnentafel I do my best to organize photos into the family/generation that is most represented by the photo and then into "events" if possible. I then use lightroom to tag the photos with as much information that I can, especially the names of the people in the photos. This way I figure, even the least technical person can find photos from a specific family but with slightly more effort, they can also search for people across the folder hierarchy.
  • im3w1l 5y ago
    As far as future proof goes, Danbooru has been on the web for 15 years and on github for 10 and is still actively developed. There is a web interface and also a webapi. But the (lack of) access control may be a problem. Maybe with appropriate nginx auth config you can ensure only the family can access it. If you search for it beware NSFW.
  • ClumsyPilot 5y ago
    "I don't trust any tagging solution to be useable in 5+ years" Damn me, but this is true. Our industry is totally irresponsible and run by children.
  • eredengrin 5y ago
    I've been meaning to try out PhotoStructure[1] for a while now as it looks like it might just be a good place to finally put some effort into organizing my collection. Nothing is guaranteed to last forever but it looks like the author is planning ahead in this case such that even if he goes out of business, all the work I put into organizing would not be meaningless (data is stored in a sqlite db and code will be open sourced in event of business closure). [1] https://photostructure.com/ https://photostructure.com/
• zmix 5y ago
  macOS had this since Leopard days. First it was implemented via the OpenMeta([1],[2]) and later, by Apple, via Maverick meta tags. They both use the XATTRS of the filesystem to store the meta-data. I didn't use macOS for a long time now, but when OpenMeta was still active, there were tons of applications, that supported it. I loved it and miss it badly. The only issue I had with it, was adding all those many tags to your files. Something, I'd like to solve with AI, these days. For me, the best system is a hierarchical file system with meta tags in XATTRS, that also can be used to build hierarchies, if needed. [1]: https://code.google.com/archive/p/openmeta/ https://code.google.com/archive/p/openmeta/ [2]: https://www.google.com/search?q=openmeta+macos https://www.google.com/search?q=openmeta+macos
• imglorp 5y ago
  https://perkeep.org/ https://perkeep.org/ seems relevant and not mentioned yet here.
• deleted 5y ago
  [deleted]
• 1970-01-01 5y ago
  Tagging assumes user will always have the ability to type or input the file tag metadata. Hierarchy allows finding files by just clicking up or down the hierarchy tree and double clicking the file(s).
• deleted 5y ago
  [deleted]
• malkia 5y ago
  I wonder how, a typical #include "someLib/someLib2/someThing.h" would look like in a "tagging" world? Sorry, probably stupid example, but hierarchial has been there from the begining (I still remember how limiting Apple ][ DOS was without folders, and how cool ProDos when it introduced them).
  • ReleaseCandidat 5y ago
    > #include "someLib/someLib2/someThing.h" would look like in a "tagging" world #include "someLib/someLib2/someThing.h" Or whatever you choose as a delimiter for a hierarchy of tags. The biggest advantage is that you can use an (well ;) infinite number of tags to get to the same 'file' (without the hassle of needing to use symlinks), so that could also be #include "currentlyUsedLibs/someThing.h" and #include "foo/bar/baz/what/why/hmm/someThing.h" The problem isn't the hierarchy, but that there only exist a single (modulo symlinks) hierarchy to every 'file'. And that every sibling must have a distinct 'name' (the cat images in the article).
    • malkia 5y ago
      There is also the pecularity in the C++ preprocessor that it'll probe candidates among -I<folders> and pick first one that suceed. Now you would need to support this for "tagging" too. Not sure even how. It does not translate well I think. (Obviously I'm just complaining here, on the side of artists I think it was much easier for them (and for me actually, when comes to my own photos) to find things by tags - be it automatical, date, time, is it a document, location, etc.) - but then having to rely on consistent ("stable") name is going to be hard.
• nathias 5y ago
  depends on what you are doing, but when you are implementing abstraction folder hierarchies are a very good analogy of concept containment
  • peterhil 5y ago
    And then we should talk about categories (and subcategories, relations or associations) instead of tags, which to many people seem to represent a flat namespace of tags.
• BiteCode_dev 5y ago
  Hierarchies and tags are not an "xor" proposition. I always use both at the same time. Hierarchy gives you a way to organize things according to each others. I want hierarchy, because if I deal with thing A, I may have then to specifically have a look to child B or C. It helps with context and granularity. While tags allow you to attach multiple categories to things and filter according to that. You can use a hierarchy as a poor man tag, but really, you should use both.
• joshu 5y ago
  Finally, a topic I can speak with some authority on. I'm not sure that tags make that much sense as an organizing schema for a filesystem. It's just too painful for items to not have canonical names. In my original design, a bookmark had a bunch of tags but was keyed on it's canonical URL. Maybe it makes more sense when we're only talking about a user's personal objects rather than the entire filesystem. Having a giant, flat namespace also seems wrong - Wikipedia seems especially strange in this regard, having Thingy_(Star_Wars) where Thingy is both a real thing and also present in some specific context; I frequently think it should be Star_Wars/Thingy (since it has a specific context it makes sense in and does not make sense outside of it.) Tags were mostly a way for a user to mark found things for that user's future self to potentially retrieve. They end up encoding some combination of the user's internal state and the material of the item being tagged. Different users therefore have different needs - expert users tag things differently than beginners (ie "java" vs "programming" - to a beginner there's little distinction between subtypes of programming things.) Users tend to tag much more wildly when they are not offered a reference of previously-used tags, as well. In my experience, an item that had been tagged a large number of times would have the most popular tag used 50% of the time, the second 33% of the time, and so on. This held strongly enough that much deviation from this was a sure sign of spamming. This is also where tagging falls down - tagging things for retrieval by OTHER people is almost always mis-incentivized and ands up getting spammed to hell.
• mikewarot 5y ago
  If you're going to use a non-standard tagging system, it's best to have a backup. For example, Gmail uses tags instead of folders, and to most uses they seem to be folders. It's possible to not even be aware of this difference. Sometimes gmail loses all your tags. Could you imagine if all your files ended up in the root? Also consider how you would back up such a schema in a traditional file system.
• ptspts 5y ago
  Shameless plug: local file tagging software for Linux, which stores tags in extended attributes: https://github.com/pts/ppfiletagger https://github.com/pts/ppfiletagger
• zuj 5y ago
  Ah, it's that time of the year again. I see this blogpost shared and discussed year after year.
• leoc 5y ago
  Here I repost a composite of my old comments https://news.ycombinator.com/item?id=14542595 https://news.ycombinator.com/item?id=14542595 https://news.ycombinator.com/item?id=14546682 https://news.ycombinator.com/item?id=14546682 from a previous occasion when this was discussed https://news.ycombinator.com/item?id=14537650 https://news.ycombinator.com/item?id=14537650 . I confess to a certain amount of annoyance that the author of this document asked for feedback on this website, but after getting that feedback didn't (afaics) update the site to make any mention of Namesys or ReiserFS/Reiser4. > Well, since you ask, here's Hans Reiser's old stuff: https://reiser4.wiki.kernel.org/index.php/Future_Vision https://reiser4.wiki.kernel.org/index.php/Future_Vision https://reiser4.wiki.kernel.org/index.php/V4 https://reiser4.wiki.kernel.org/index.php/V4 (and http://lwn.net/2001/1108/a/reiser4-transaction.php3 http://lwn.net/2001/1108/a/reiser4-transaction.php3 ) . And here's some emails etc. I wrote in response: https://web.archive.org/web/20040728044342/http://www.st-andrews.ac.uk/~lrc1/pathname_metadata.html https://web.archive.org/web/20040728044342/http://www.st-and... https://marc.info/?l=linux-kernel&m=111624697710426 https://marc.info/?l=linux-kernel&m=111624697710426 https://www.mail-archive.com/reiserfs-list@namesys.com/msg09880.html https://www.mail-archive.com/reiserfs-list@namesys.com/msg09... https://www.mail-archive.com/reiserfs-list@namesys.com/msg20060.html https://www.mail-archive.com/reiserfs-list@namesys.com/msg20... https://www.mail-archive.com/reiserfs-list@namesys.com/msg20061.html https://www.mail-archive.com/reiserfs-list@namesys.com/msg20... https://www.mail-archive.com/reiserfs-list@namesys.com/msg20063.html https://www.mail-archive.com/reiserfs-list@namesys.com/msg20... https://www.mail-archive.com/reiserfs-list@namesys.com/msg20214.html https://www.mail-archive.com/reiserfs-list@namesys.com/msg20... , plus some of the discussion threaded from those posts. (Sorry, my stuff needs rewriting and updating but I'm not in the position to do it at present. If there's anything you would like to ask about please do. https://news.ycombinator.com/item?id=9809041 https://news.ycombinator.com/item?id=9809041 and https://news.ycombinator.com/item?id=10548477 https://news.ycombinator.com/item?id=10548477 touch on things that are a bit further down the line, but related—in particular, to the handling of "internal metadata" and files with a compound internal structure.)
• flohofwoe 5y ago
  I've been thinking about this a lot in the past and also think that tags would indeed make the most sense, at least for a new type of "desktop UI" which is created around the idea to quickly listing and finding files by typing tag names instead of an actual "desktop metapher". I guess the idea didn't get much traction because coders are often using the cmdline (where the hierarchical organization works reasonably well), while creative professionals often use the file organization methods offered in specific applications and probably don't spend all that much time in Finder/Explorer (except maybe for bulk-copying files around).
• jsilence 5y ago
  If only the HFS and the OS using it would consistently yield the documents I recently used, across all file system dialogues, that alone would be a huge benefit for me. Apart from that I'd love to see a tag based file system. Would use.
• jFriedensreich 5y ago
  git annex gets very close to what he had in mind.
• ChrisArchitect 5y ago
  Anything new here? Lots of previous discussion: https://news.ycombinator.com/item?id=16763235 https://news.ycombinator.com/item?id=16763235
• bacan 5y ago
  What are folders, if not tags with nesting capabilities? Not like they physically exist on the disk beyond some text & file data
• peterhil 5y ago
  The article fails to mention existence of graph databases or graph theory, except for tuple spaces which leads to that direction. Property graph model is the keywords to search for if you are interested in graph databases' suitability for data organisation: http://graphdatamodeling.com/Graph%20Data%20Modeling/GraphDataModeling/page/PropertyGraphs.html http://graphdatamodeling.com/Graph%20Data%20Modeling/GraphDa... With graph databases you can easily and efficiently model any kind of network, including ones that are hierarchical or almost hierarchical by allowing a node to refer to multiple parent nodes instead of one. My original idea with my bookmark extension Spellbook was this latter kind of graph, and I implemented a prototype called Grimoire using Ruby on Rails and Neo4j graph database that worked very well. The Spellbook currently only allows adding new bookmarks into the hierarchical structure imposed by browser APIs, but features an easy to use search feature to find the right category. Spellbook is available for Chrome and Firefox, but the Firefox version seems broken again by their API changes: https://github.com/peterhil/spellbook https://github.com/peterhil/spellbook
  • peterhil 5y ago
    I got the idea for Spellbook, because I was learning programming, but also having some projects that involved quite a bit of research on subjects like audio physics, statistics, data visualisation etc., and I wished I could keep my bookmarks in a generic hierarchy, but also attach some categories (bookmark folders) to categories for my different projects at the same time. So, I also see the value in having subcategories in addition to freely formed networks or associations.
  • peterhil 5y ago
    I also had the idea to apply these kind on ideas about nonhierarchical file organisation to music or photo library organisation, and really wish there was a file system level support.