4 ms·

Designing better file organization around tags, not hierarchies (2017)

43 points by Tomte 1y ago

Telemakhos 1y ago
This argument's been coming up every now and then since at least 2010, but it never goes anywhere. macOS has had tags in Finder for years, and few people actually use them; I like them and still fail to use them consistently. I wonder if there's a counter-argument to be made that humans organize their knowledge of the world in ontological hierarchies, so a hierarchical file system is intuitive.
- bonoboTP 1y ago
  Many years ago I tried Tabbles (https://tabbles.net/en/ https://tabbles.net/en/), apparently it still exists, but it just didn't work out so well. The main concern is compatibility. I tag up my stuff meticulously and then the project is killed and I'm left with wasted effort. I think the idea will go away, similar to how ontologies and Semantic Web, manual knowledge graphs etc gave way to processing unstructured data with LLMs. Instead of tags, we should have source/creation-context-based metadata and embeddings computed with language models. Then you can do natural fuzzy search.
- taeric 1y ago
  I think the main counter-argument is that people like to organize in static systems? Even if you use tags, you almost certainly want to restrict the specific tags that you allow so that you don't have a data science/cleanup task of normalizing things later. Tags also fail because people then want to categorize their tags. Which tag is the author's name, versus the editor's? Publish year, versus year I read it? I suppose we could say that people want "slots" not "tags"?
  - esafak 1y ago
    And mayhem ensues if other people want to have a say in the ontology.
comrade1234 1y ago
There's a standard used in content management systems that's been around for, I think, over twenty years - JCR, or Java Content Repository. I know this is about an OS and not a CMS, but it seems storing your documents in a repository like this would be useful - not sure about the rest of the OS though.
kstrauser 1y ago
(HFS here means “hierarchical file system”, i.e. one with nested folders.) I tire of the either-or dichotomy. Yes, search is great. But a couple years ago, I adopted Johnny Decimal[0] for storing my archive files and I couldn’t be happier. Our 2023 tax filings are in 22.23. Physics articles are in 34.04. Her CME records are in 43.03. Go into one of those and you’ll find every file I have related to those things. And they’re still searchable. You can still do tag-based queries, indexed text searches, and all that, but you don’t have to. When I want all my 2025 tax deduction receipts come tax filing time next year, I’ll send the contents of 22.25 to my accountant and be done with it, confident we found 100% of the related files. I don’t do those for everything. Music files are in their own directories. Movies live with movies. I don’t try to shoehorn Obsidian into it. But all my long-term storage files go straight to their home in JD. This has the huge benefit that I can instantly locate them when I’m accessing the archive from my phone, my wife can find things without learning special tooling or the vagaries of search, and there’s zero need for any additional apps or software beyond Finder. I love using great search software. I used Devonthink daily for years. But I also adore not having to. [0] https://johnnydecimal.com/ https://johnnydecimal.com/
- taeric 1y ago
  I can only assume this is at least a spiritual ode to Dewey Decimal? Looks neat. I had hoped you meant that not being "either-or" would be that you should be able to get "realized hierarchies" on things. Where you could basically ask the computer to build a hierarchy on the fly.
  - kstrauser 1y ago
    Huh, that’s an interesting idea. “Hey AI, here are 10,000 files. Make a hierarchical categorization of them” sounds like fun. In my case, I got a lot of value in coming up with that on my own. The result may not be rigorously perfect. In fact, I know it isn’t: some folders have lots of files, and some have few. It maps really well to my brain, though. That makes it easier for me to use, because I’m not memorizing a hierarchy someone else invented and pushed upon me. I’m using the system that sounded about right to me in the first place.
ednite 1y ago
From what I understand, the author's proposal for a tag-based file system reminds me of how Microsoft Azure Blob Storage works. Azure blobs support multiple queryable tags per file, which aligns with the idea of organizing by metadata instead of folders. It’s not as deeply integrated or user-centric as the article suggests, but it offers a practical usage, especially for large-scale, developer-driven storage.
lizknope 1y ago
I'm going to admit that I didn't read the whole article. I remember about 10-15 years ago that there were a lot of photo organizers that supported tagging. Tag the events as birthday or the people's names. I saw a lot of people who would then post "My photo organizer has been discontinued and I spent years organizing by tags. How do I migrate all the tags to another photo editor." The hierarchical file system is kind of a lowest common denominator. I can copy a directory structure from ext4 to brtfs to zfs to ntfs to exfat to macos or whatever and have the same organization.
- esafak 1y ago
  DAM is the word! (Digital asset management.)
- 0cf8612b2e1e 1y ago
  This is exactly why I cannot get on board with tagging. Without a universal standard, I am one deprecated app away from losing my time investment.
  - Lammy 1y ago
    > Without a universal standard The universal standard is Dublin Core (IETF RFC 2413 / ISO 15836) `dc:subject` “Unordered array of Text” stored in an XMP (ISO 16684) sidecar file: - https://en.wikipedia.org/wiki/Dublin_Core#Dublin_Core_Metadata_Element_Set,_1995 https://en.wikipedia.org/wiki/Dublin_Core#Dublin_Core_Metada... - https://en.wikipedia.org/wiki/Extensible_Metadata_Platform https://en.wikipedia.org/wiki/Extensible_Metadata_Platform I like to put the DC stuff directly in my filesystem xattrs, but using a sidecar file will get you compatibility with any old filesystem and with any tooling that might destroy or disregard an xattr.
    - jeffbee 1y ago
      This is an amazing bit already, but it would be a 10/10 joke if you could work "year of the Linux desktop" into it somewhere. If you were serious, then I would point out that even nerd file browsers like Nautilus do not support xattr metadata editing, searching, or even viewing and Linux metadata indexing systems e.g. Beagle are all long-dead projects. The whole idea has no traction.
skydhash 1y ago
Tags only work when it's a restricted domain. Like your music files, or ebooks, or your pictures library. So the best way is to construct multiple volumes, one for each type of repository. But it's not truly universal because of hardware devices you can't program as we don't have standards for tags.
touwer 1y ago
It could still be a good idea. Wrote about almost the same thing in 2008 https://wouter.info/tag_based_file_management.pdf https://wouter.info/tag_based_file_management.pdf
Seattle3503 1y ago
Are there good examples tools for tagging files? I've long been interested,but haven't found anything good.
esafak 1y ago
I wouldn't bother; the time for this is past. It is not worth curating tags/ontologies. Let the computer do it for you, or don't: just search. Google was right. The problem is acute in organizations, where the question of ontology ownership crops up. The last place I worked at had a dedicated engineer on the job. And marvel at the elaborate systems Wikipedia and StackOverflow have to manage their ontologies. It is best to avoid that stuff and improve search instead, which is likely why an ontology was sought in the first place.
- theptip 1y ago
  It’s a good observation. Counterpoint - the organization labor is drudge work, maybe with cheap AI labor you can have your KB kept in great shape, and likely this makes it easier for RAG to locate the correct document. Maybe there is room for both, though I am in agreement that it seems dubious a human should be curating.
  - Spivak 1y ago
    I mean that's basically what all these image search indexes do. They look at an image and add a bunch of tags for whatever they see in the photo that can be queried later. It works really well and armed with one of these models and word2vec you can build your own search quite easily. tensorflow object detection is one such means. Hosted solutions would be Amazon Rekognition.
- nabla9 1y ago
  > Let the computer do it for you, or don't: just search. Only for information that has no real value. Search misses things. Hierarchy or tag ensures that all you have marked can be found.
  - esafak 1y ago
    Like what, domain-specific information? If so, that's a question of fine tuning the (machine learning) model. Thereafter, it can automatically tag as it indexes.
rmah 1y ago
The primary problems with tag-based organization are that 1) tags are difficult to organize hierarchically and 2) discoverability of existing tags tends to be poor. However, if you have multiple overlapping hierarchical organization, common operations such as "moving" and "deleting" become overly complicated. IMO, there's no best way, only different pros/cons and compromises.
taco_emoji 1y ago
I still want hierarchies. Tags would be a great addition, though.
mattmcknight 1y ago
the one thing about hierarchies that keeps them alive for me is that I am forced to assign one when I save a file, and they already exist. Sure, you could have a dumping ground like a "downloads" folder, but that's temporary or expendable. It's probably easier to just through everything into a clustering algorithm and autotag than going back and tagging tens of thousands of files.
kuharich 1y ago
Past comments: https://news.ycombinator.com/item?id=29141800 https://news.ycombinator.com/item?id=29141800
kkfx 1y ago
My files are mostly org-mode/org-attach-ed stuff, linked in notes so they physically reside in a cache-like structure, accessed via search&narrow. Classic taxonomies are a thing of the past.
porridgeraisin 1y ago
Actually I don't care for any kind of organization for personal files. I just search. I have my _really_ important files that are cumbersome to recreate like a <50kb jpg of my signature for those annoying forms, scans of my identity cards, etc, in ~/Documents/important I have my to-read stuff in ~/Documents/books All my other files are slammed together in ~/Downloads. I search through the folder using everything.exe (on windows) or fsearch(on linux). That already covers the 90% case, and for the rest recoll maintains a search index that even stores file contents (including zip,pdf,xml-based stuff like office files). It works well. These days I hear everything.exe itself supports searching within files. On the other hand, I'm really picky about properly hierarchically organizing code files in any SW project I do, and proper hierarchical organizing of the app folder of any app I make. As for tags, the only reason I see myself using them is for pictures. It can enable a deterministic version of Google photos search. But I'm happy with Google photos search for now.

43 points by Tomte 1y ago

This argument's been coming up every now and then since at least 2010, but it never goes anywhere. macOS has had tags in Finder for years, and few people actually use them; I like them and still fail to use them consistently. I wonder if there's a counter-argument to be made that humans organize their knowledge of the world in ontological hierarchies, so a hierarchical file system is intuitive.

bonoboTP 1y ago
Many years ago I tried Tabbles (https://tabbles.net/en/ https://tabbles.net/en/), apparently it still exists, but it just didn't work out so well. The main concern is compatibility. I tag up my stuff meticulously and then the project is killed and I'm left with wasted effort. I think the idea will go away, similar to how ontologies and Semantic Web, manual knowledge graphs etc gave way to processing unstructured data with LLMs. Instead of tags, we should have source/creation-context-based metadata and embeddings computed with language models. Then you can do natural fuzzy search.
taeric 1y ago
I think the main counter-argument is that people like to organize in static systems? Even if you use tags, you almost certainly want to restrict the specific tags that you allow so that you don't have a data science/cleanup task of normalizing things later. Tags also fail because people then want to categorize their tags. Which tag is the author's name, versus the editor's? Publish year, versus year I read it? I suppose we could say that people want "slots" not "tags"?
- esafak 1y ago
  And mayhem ensues if other people want to have a say in the ontology.

comrade1234 1y ago

There's a standard used in content management systems that's been around for, I think, over twenty years - JCR, or Java Content Repository. I know this is about an OS and not a CMS, but it seems storing your documents in a repository like this would be useful - not sure about the rest of the OS though.

kstrauser 1y ago

(HFS here means “hierarchical file system”, i.e. one with nested folders.) I tire of the either-or dichotomy. Yes, search is great. But a couple years ago, I adopted Johnny Decimal[0] for storing my archive files and I couldn’t be happier. Our 2023 tax filings are in 22.23. Physics articles are in 34.04. Her CME records are in 43.03. Go into one of those and you’ll find every file I have related to those things. And they’re still searchable. You can still do tag-based queries, indexed text searches, and all that, but you don’t have to. When I want all my 2025 tax deduction receipts come tax filing time next year, I’ll send the contents of 22.25 to my accountant and be done with it, confident we found 100% of the related files. I don’t do those for everything. Music files are in their own directories. Movies live with movies. I don’t try to shoehorn Obsidian into it. But all my long-term storage files go straight to their home in JD. This has the huge benefit that I can instantly locate them when I’m accessing the archive from my phone, my wife can find things without learning special tooling or the vagaries of search, and there’s zero need for any additional apps or software beyond Finder. I love using great search software. I used Devonthink daily for years. But I also adore not having to. [0] https://johnnydecimal.com/ https://johnnydecimal.com/

taeric 1y ago
I can only assume this is at least a spiritual ode to Dewey Decimal? Looks neat. I had hoped you meant that not being "either-or" would be that you should be able to get "realized hierarchies" on things. Where you could basically ask the computer to build a hierarchy on the fly.
- kstrauser 1y ago
  Huh, that’s an interesting idea. “Hey AI, here are 10,000 files. Make a hierarchical categorization of them” sounds like fun. In my case, I got a lot of value in coming up with that on my own. The result may not be rigorously perfect. In fact, I know it isn’t: some folders have lots of files, and some have few. It maps really well to my brain, though. That makes it easier for me to use, because I’m not memorizing a hierarchy someone else invented and pushed upon me. I’m using the system that sounded about right to me in the first place.

ednite 1y ago

From what I understand, the author's proposal for a tag-based file system reminds me of how Microsoft Azure Blob Storage works. Azure blobs support multiple queryable tags per file, which aligns with the idea of organizing by metadata instead of folders. It’s not as deeply integrated or user-centric as the article suggests, but it offers a practical usage, especially for large-scale, developer-driven storage.

lizknope 1y ago

I'm going to admit that I didn't read the whole article. I remember about 10-15 years ago that there were a lot of photo organizers that supported tagging. Tag the events as birthday or the people's names. I saw a lot of people who would then post "My photo organizer has been discontinued and I spent years organizing by tags. How do I migrate all the tags to another photo editor." The hierarchical file system is kind of a lowest common denominator. I can copy a directory structure from ext4 to brtfs to zfs to ntfs to exfat to macos or whatever and have the same organization.

esafak 1y ago
DAM is the word! (Digital asset management.)
0cf8612b2e1e 1y ago
This is exactly why I cannot get on board with tagging. Without a universal standard, I am one deprecated app away from losing my time investment.
- Lammy 1y ago
  > Without a universal standard The universal standard is Dublin Core (IETF RFC 2413 / ISO 15836) `dc:subject` “Unordered array of Text” stored in an XMP (ISO 16684) sidecar file: - https://en.wikipedia.org/wiki/Dublin_Core#Dublin_Core_Metadata_Element_Set,_1995 https://en.wikipedia.org/wiki/Dublin_Core#Dublin_Core_Metada... - https://en.wikipedia.org/wiki/Extensible_Metadata_Platform https://en.wikipedia.org/wiki/Extensible_Metadata_Platform I like to put the DC stuff directly in my filesystem xattrs, but using a sidecar file will get you compatibility with any old filesystem and with any tooling that might destroy or disregard an xattr.
  - jeffbee 1y ago
    This is an amazing bit already, but it would be a 10/10 joke if you could work "year of the Linux desktop" into it somewhere. If you were serious, then I would point out that even nerd file browsers like Nautilus do not support xattr metadata editing, searching, or even viewing and Linux metadata indexing systems e.g. Beagle are all long-dead projects. The whole idea has no traction.

skydhash 1y ago

Tags only work when it's a restricted domain. Like your music files, or ebooks, or your pictures library. So the best way is to construct multiple volumes, one for each type of repository. But it's not truly universal because of hardware devices you can't program as we don't have standards for tags.

touwer 1y ago

It could still be a good idea. Wrote about almost the same thing in 2008 https://wouter.info/tag_based_file_management.pdf https://wouter.info/tag_based_file_management.pdf

Seattle3503 1y ago

Are there good examples tools for tagging files? I've long been interested,but haven't found anything good.

esafak 1y ago

I wouldn't bother; the time for this is past. It is not worth curating tags/ontologies. Let the computer do it for you, or don't: just search. Google was right. The problem is acute in organizations, where the question of ontology ownership crops up. The last place I worked at had a dedicated engineer on the job. And marvel at the elaborate systems Wikipedia and StackOverflow have to manage their ontologies. It is best to avoid that stuff and improve search instead, which is likely why an ontology was sought in the first place.

theptip 1y ago
It’s a good observation. Counterpoint - the organization labor is drudge work, maybe with cheap AI labor you can have your KB kept in great shape, and likely this makes it easier for RAG to locate the correct document. Maybe there is room for both, though I am in agreement that it seems dubious a human should be curating.
- Spivak 1y ago
  I mean that's basically what all these image search indexes do. They look at an image and add a bunch of tags for whatever they see in the photo that can be queried later. It works really well and armed with one of these models and word2vec you can build your own search quite easily. tensorflow object detection is one such means. Hosted solutions would be Amazon Rekognition.
nabla9 1y ago
> Let the computer do it for you, or don't: just search. Only for information that has no real value. Search misses things. Hierarchy or tag ensures that all you have marked can be found.
- esafak 1y ago
  Like what, domain-specific information? If so, that's a question of fine tuning the (machine learning) model. Thereafter, it can automatically tag as it indexes.

rmah 1y ago

The primary problems with tag-based organization are that 1) tags are difficult to organize hierarchically and 2) discoverability of existing tags tends to be poor. However, if you have multiple overlapping hierarchical organization, common operations such as "moving" and "deleting" become overly complicated. IMO, there's no best way, only different pros/cons and compromises.

taco_emoji 1y ago

I still want hierarchies. Tags would be a great addition, though.

mattmcknight 1y ago

the one thing about hierarchies that keeps them alive for me is that I am forced to assign one when I save a file, and they already exist. Sure, you could have a dumping ground like a "downloads" folder, but that's temporary or expendable. It's probably easier to just through everything into a clustering algorithm and autotag than going back and tagging tens of thousands of files.

kuharich 1y ago

Past comments: https://news.ycombinator.com/item?id=29141800 https://news.ycombinator.com/item?id=29141800

kkfx 1y ago

My files are mostly org-mode/org-attach-ed stuff, linked in notes so they physically reside in a cache-like structure, accessed via search&narrow. Classic taxonomies are a thing of the past.

porridgeraisin 1y ago

Actually I don't care for any kind of organization for personal files. I just search. I have my _really_ important files that are cumbersome to recreate like a <50kb jpg of my signature for those annoying forms, scans of my identity cards, etc, in ~/Documents/important I have my to-read stuff in ~/Documents/books All my other files are slammed together in ~/Downloads. I search through the folder using everything.exe (on windows) or fsearch(on linux). That already covers the 90% case, and for the rest recoll maintains a search index that even stores file contents (including zip,pdf,xml-based stuff like office files). It works well. These days I hear everything.exe itself supports searching within files. On the other hand, I'm really picky about properly hierarchically organizing code files in any SW project I do, and proper hierarchical organizing of the app folder of any app I make. As for tags, the only reason I see myself using them is for pictures. It can enable a deterministic version of Google photos search. But I'm happy with Google photos search for now.