8 ms·

I am convinced that absolutely no hierarchical taxonomy actually works in practice because every categorisation is really slightly fuzzy. Even libraries have pr

by baz00 3y ago

I am convinced that absolutely no hierarchical taxonomy actually works in practice because every categorisation is really slightly fuzzy. Even libraries have problems categorising books into genres because some of them overlap considerably. I spent the last 25 years trying to build up useless hierarchies everywhere and none of them ever worked properly. Tagging/labelling with attributes is the only viable solution that I've found. This book is fiction/science-fiction/fantasy. This document is technical-reference/specification/ratified I believe this was the notion of the now defunct WinFS built on top of a database engine.

ThouYS 3y ago
this, tags + dynamic views on those tags
logdahl 3y ago
I'm just wondering when we're gonna get tagged filesystems. Maybe something partially hierarchical. Would love a book/ directory that i can query for 'sci-fi' or 'specification'. I guess one could implement it using sym-links :^)
- baz00 3y ago
  Don't even go there with the symlinks. My mother made a family tree on windows with folders and shortcuts. It broke OneDrive and robocopy completely. I don't even know how to help her fix it.
  - virtue3 3y ago
    It's 3 in the morning and I couldn't stop laughing out loud when I read this. Poor OneDrive lol. I hope she managed to keep all her files and didn't lose anything.
    - baz00 3y ago
      It still works and she hasn't lost anything but you can't actually move or delete any files any more because windows explorer craps out too! I literally have no idea what to do with it :(
      - hobs 3y ago
        My assumption is she ran into the max path length limitation, here's how to fix it https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
        baz00 3y ago
        That was one of the problems. Unfortunately after fixing that it's still broken.
  - alpaca128 3y ago
    On one hand this is an edge case, on the other things like this make me wonder how computers can even work if first-party software built specifically for nothing but managing files fails to gracefully handle a folder structure a user created by hand.
    - baz00 3y ago
      That was exactly my take home from this. Windows has some very serious flaws in it when it comes to data integrity. I do not trust it and do not use it myself.
  - mceachen 3y ago
    You could write a short PowerShell script that recursively deletes shortcuts and symlinks, but I'd certainly take a full backup and have it offline before I kicked out off.
- theshrike79 3y ago
  There are some attempts using Fuse on Linux. At least one I tried ages ago was one you could use to browse your music collection, it automatically generated "directories" based on mp3tags.
- Brajeshwar 3y ago
  Apple tried doing this with "tag" in the file system but most people don't use it.
  - baz00 3y ago
    I use it! :)
    - Brajeshwar 3y ago
      Don’t Do That. Don’t Give Me Hope.
    - smallerdemon 3y ago
      Same.
  - hnbad 3y ago
    Microsoft allegedly wanted to get rid of the folder hiearchy completely. There were a lot of rumors surrounding "Windows 97" (what later became Windows 98) because of its Internet integration (which eventually just turned into a lot of dead-end MSN widgets and Internet Explorer being shipped with the OS). I don't know how well-founded this speculation was but allegedly Microsoft wanted to fully commit to things like "cloud storage" (before this was a thing, so of course not using this name) and ended up shelving most of this work because the networks just weren't fast enough at the time. Note: I specifically remember these speculations in the lead-up to the release of Windows 98. They don't seem to be easy to google at this point.
- timschmidt 3y ago
  May 10, 1997: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System
- pedrovhb 3y ago
  I just wish xattrs were more resilient and explicit.
- Cyphase 3y ago
  The author of TMSU left a sibling comment to yours: https://news.ycombinator.com/item?id=37507343 https://news.ycombinator.com/item?id=37507343 https://tmsu.org/ https://tmsu.org/ > TMSU is a tool for tagging your files. It provides a simple command-line tool for applying tags and a virtual filesystem so that you can get a tag-based view of your files from within any other program. > TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever you put them. TMSU maintains its own database and you simply gain an additional view, which you can mount, based upon the tags you set up. The only commitment required is your time and there's absolutely no lock-in.
  - notfed 3y ago
    Got me wondering how it tracks file moves. From the FAQ: > TMSU [does] not automatically detect file moves and renames I dunno if we can call it a tagged filesystem with this limitation. (Why not store tags, or at least a GUID, in file metadata?)
    - istjohn 3y ago
      It does store file hashes. It has a repair command that will use file hashes to find moved files. I don't know how well it works.
    - dotancohen 3y ago
      This seems a serious limitation. Bash is my file manager - ls and cd. I would not give that up for any tool no matter how convenient. But presumably there is some API or ABI to monitor, or some signal emitted, when files are moved, added, updated, or deleted. However iwaitnotify works, could that be incorporated (even theoretically) into TMSU?
      - Cyphase 3y ago
        https://github.com/oniony/TMSU/wiki/FAQ#why-does-tmsu-not-automatically-detect-file-moves-and-renames https://github.com/oniony/TMSU/wiki/FAQ#why-does-tmsu-not-au... There are a couple very barebones wrappers around mv and rm, though they could be better (pass through arguments, etc.). https://github.com/oniony/TMSU/wiki/Tricks-and-Tips#filesystem-operations https://github.com/oniony/TMSU/wiki/Tricks-and-Tips#filesyst...
- dredmorbius 3y ago
  The approach I've been favouring for a while has been to have a data store, I call it the "stacks", based off physical library nomenclature, and then a virtual filesystem which effectively uses searches along various attributes to surface individual items. Those searches might be based on title, author/creator/contributor, publication/creation dates, assigned identifiers, full-text search, relations between documents, subjects or topics, document type / format, or other aspects. How that is specified precisely ... I haven't settled on entirely, though short mnemonics, two- or three-letter where possible (ti -> title, au -> author, date, isbn, oclc, doi, etc.) might work. If the filesystem lives under /docfs, then, say, /docfs/au:barrie/ti:peter+pan would turn up J.M. Barrie's Peter Pan. Specific document identifiers such as ISBNs, OCLCs, DOIs, or LCCSs could pinpoint specific documents, looser searches such as for Peter Pan might generate a list rather than a unitary response but those individual documents would be further distinguished by other properties. The filesystem would be virtual rather than static-on-disk. You're effectively exploring the namespace. Creating this as a filesystem rather than, say, as a database query or application-specific data store means that standard filesystem tools would be available to interact with the contents, though modifications of works might require some additional work. The system could also provide access to works not directly in the stacks, including remote resources (e.g., accessed via URIs and network calls), or applications (through some API). The underlying data store could be in any of several formats, and again the filesystem-based access could abstract between several independent stores. Now all I need to do is build it ...
  - Cyphase 3y ago
    This sounds at least somewhat like TMSU, which has been mentioned in some other comments here. https://tmsu.org/ https://tmsu.org/
    - dredmorbius 3y ago
      I've seen that, yes. There are some similarities, and I'm thinking that TMSU might serve as a component of this system.
pbmonster 3y ago
> Tagging/labelling with attributes is the only viable solution that I've found. What's your solution for a new tag being introduced late, which could/should be retroactively applied to already tagged media? Re-tag the entire inventory by hand? Re-tag automatically with some ML/AI tool after you've tagged enough media to train a discriminator? Or just live with old inventory not having that tag?
- baz00 3y ago
  Depends very much on the problem domain. It's really easy to do this on macOS. So for example I have a top level `Documents/docs` directory. In this everything is recorded in format `yyyy-mm-dd Title.ext` format. I have a saved search which says "give me everything untagged". I will select each thing that is untagged and then tag it!
  - plagiarist 3y ago
    I think they're asking something different, what if you think of a brand new tag after you have already tagged things for years? It is potentially a lot of effort to retroactively apply it.
    - smallerdemon 3y ago
      Yeah. All systems like this have to account for ad-hoc additions. Otherwise you will often jam a few squares into some circles just to accommodate your categorization system.
    - cma 3y ago
      AI/LLMs can probably make it easy.
    - dvas 3y ago
      I ran into a similar situation a while back now, I have my version of a zettelkasten system, where I put a tag/ list of tags at the top of the note file (timestamp being the note name). Combining this with something like ripgrep allows me to search the note file contents by those tags, and keywords. A tag I used, didn't seem like the right one for the content as it matured and grew over time. This meant creating a tiny script which would take the name of the tag and the note name, in which it is being used and look at backlinks and forward links and rename the tag based on usage.
imrehg 3y ago
That's what I took away from reading Everything is Miscellaneous a long while back https://en.wikipedia.org/wiki/Everything_Is_Miscellaneous https://en.wikipedia.org/wiki/Everything_Is_Miscellaneous Definitely thinking about categorization differently (folders which are similar to this, tags, organic order by linking; bookmarks, notes, all are in the same boat as well).
- lukasgraf 3y ago
  I saw David Weinberger's talk at Google [1] some 15 years ago, where he presented the ideas from his book (and bought the book afterwards). This is one of those videos I rewatch regularly, because the ideas hold up so well, even today. His critique of the Dewey Decimal system was the first thing that came to mind when I read Johnny Decimal. And indeed, it is flawed the same way. [1] https://www.youtube.com/watch?v=x3wOhXsjPYM https://www.youtube.com/watch?v=x3wOhXsjPYM
rpastuszak 3y ago
This is precisely the reason why smaller cross-linked notes and tags work quite well for me, and JD or almost anything else in the past hasn't. The ideal for me involves hypertext + (local, private) AI-assisted search. None of this will replace reviewing/pruning your notes regularly (think: Andy Matuschak's fleeting and evergreen notes), but it minimises the friction.
oniony 3y ago
And what led me to build [TMSU](https://tmsu.org/ https://tmsu.org/).
- Cyphase 3y ago
  Thank you!
- vifon 3y ago
  TMSU is the only such system I found useful without being cumbersome. After years of trying to use Git Annex, it was refreshing that TMSU doesn't alter the files in any way, merely storing all the (meta)data out-of-band in a separate DB. These days I use TMSU via my own Emacs-based UI almost every single day, so thank you for that!
  - dredmorbius 3y ago
    Storing metadata out-of-band strikes me as key to any usable content management system within a realistically complex space. Naming schemes and directory hierarchies have some limited application, but ultimately there will be data which simply won't be shoehorned into any such system, and an externally-managed catalogue tying together disparate elements is required. (Keeping that catalogue up to date and consistent is a whole 'nother issue.) I do like the idea of a virtual filesystem in which elements are effectively search dimensions, which leads to an interesting notion that search is identity. That is, a search will produce one of three possible result sets: - Null, that is, no matches. - Plural, that is, a list of matches. - Unity, that is, one matching item. In the last case, the search providing a single result is an identity of that result. (It may not be a stable identity over time, but it is at least for the present.) Where a list is returned, the size of the list determines how usable it is, and how it is usable. Ten items can be quickly scanned to find the relevant item(s), if they exist. 100 or 1,000 items can often still be managed manually, though they'll typically take some time. Somewhere between 100 and a few thousand items, though, you're in the range where automated assessments or filtering becomes necessary. Large libraries themselves typically have tens of thousands to millions of items. The largest book collections (Library of Congress, British Library) have roughly 150 million books (or equivalents). Other records may exist in greater numbers: periodicals, financial records, databases. Facebook has reported ~5 billion items posted daily for some years now. (I suspect most of those are trivial, but that still leaves a large number of potentially non-trivial items.) Surveillance and other large-scale data collection systems may be larger still.
mihaaly 3y ago
Also if someone by some miracle is able to come up with a univocal hierarchy for a usecase or organization - that may even work throughout time and evolving situation and conditions - then collaborating with an other field or organization will have collisions or discrepancies in taxonomy and a hopeless confusion of category number mapping.
dotancohen 3y ago
I actually use a hierarchy of tags to organize my notes. The _tags_ are hierarchial, not the storage path.
- somat 3y ago
  but your storage path is already a hierarchy of tags.* *If using a unix derived filesystem.
  - dotancohen 3y ago
    I could use symlinks to refer to files, yes, but rather I'm using nonstandard #foo-bar-baz subtag syntax to refer to sections of org mode documents. I do not use real org mode tags because those are restricted to headlines. In other news, if anyone knows how to restrict a search in an org mode buffer to search only the headings, that would be very helpful! Right now I'm using an external grep wrapper, restricting to lines that begin with an asterisk.
    - somat 3y ago
      ehh... symlinks... perhaps, But I am talking about hardlinks. Having said that, I will be the first to admit. the reason hardlinks are not often used is that it is a real "here be dragons" part of the map. 10 years ago I spent a couple fun weeks categorizing all my music. my schema was something along the lines of every file was linked to artist/artist_name/song_name band/band_name/song_name song/song_name/band_name date/YYYY-MM-DD/song_name genre/genre_name/song_name playlist/playlist_name/index etc, etc then I had fun making up a small set of cli utilities to find tags, create and manage playlists etc. then A. it turns out I don't really listen to music. and B. the whole structure wanted to explode every time I would try and move it to a new host. so it probably still sits there on one of my file servers but now as a million duplicated files.
couchand 3y ago
If anything, the history of libraries show you can get really really far by applying a sytematic organizing principle, even if it's WRONG(TM). Librarians may fret that some books are hard to categorize, but library patrons successfully find stacks of books on their own every day all around the world. It's not perfect, but it's better than nothing!
- dredmorbius 3y ago
  Book classifications ultimately need to assign a book to a location. So long as that goal is achieved, the rationality or consistency of that assignment is a strongly-secondary consideration. If the book has a place, and can be located, where it happens to be relative to other related works, or unrelated ones, really doesn't matter. Searching for works based on similarities or relationships is a problem of the indexing system. I should be able to find works on a similar topic, which reference or are referenced by another, by author, by institution, perhaps by publisher, by publication date, etc. That information would be carried within a catalogue, but need not determine the location of the work within a physical archive.
PurpleRamen 3y ago
> I am convinced that absolutely no hierarchical taxonomy actually works in practice because every categorisation is really slightly fuzzy. Fuzzy is ok, not everything needs to be perfect. > Tagging/labelling with attributes is the only viable solution that I've found. Categories and tags have different use cases. Categories are the best solution, when there is only one possible location for an object, like with physical objects, which cannot appear at multiple locations at the same time. Tags make sense, if you can offer multiple entry points to reach the goal, and have the option to manage them all on a reliable level. And nothing speaks against using them both, with different valuesets. Like with digital files, you can have categories for managing the filetypes (books, videos, etc.), but then use tags to manage the content in each category. > This book is fiction/science-fiction/fantasy. Isn't this kinda redundant? science-fiction is already fiction, it's in the name. Reads more like a hierarchy. Which you of course you can also with tags. Any fantasy-tag automatically inherits the fiction-tag too.
- giaour 3y ago
  > Isn't this kinda redundant? science-fiction is already fiction, it's in the name. This is nit-picking the specific example rather than engaging with the GP's point. There are definitely genres that span the fiction/non-fiction divide (like romance, horror, politics, judaica, and many more). Interestingly, hierarchical systems used by libraries tend to ignore genre. The Dewey Decimal system doesn't classify literature by genre but instead by language, though some overarching genres (drama vs poetry) are treated as subcategories of language (e.g., German poetry is 831, French poetry is 841), but only for major European languages.
  - PurpleRamen 3y ago
    > This is nit-picking the specific example rather than engaging with the GP's point. It's showing that tags can be fuzzy too. So the initial complaint is not avoided, just moved.
    - deleted 3y ago
      [deleted]
eek04_ 3y ago
Scientific libraries categorise into *several categories*. They effectively use hierarchical tags, with the set of tags very carefully curated.
somat 3y ago
tagging is what happens when you allow an artifact to exist under more than one hierarchy at the same time. The progression usually goes. "we want to label our things." "There is a natural hierarchy that we want to be reflected in our labels. "Some things are not labeling well. instead use multiple labels per thing." "There is a natural hierarchy that we want to be reflected in our labels." The unix files system is a good example of the latter. it favors the hierarchy but allows each file to have many names if that is what is required. The main weakness of the unix filesystem is that there is no reverse map, that is, finding a file from its name(s) is simple. finding the names from the file involve searching the whole filesystem.
jjtheblunt 3y ago
I've thought a ton about what you just described, and I think of it this way: not every directed acyclic graph happens to be a tree.
tuatoru 3y ago
> Tagging/labelling with attributes is the only viable solution that I've found. The only solution I've found is being utterly ruthless about deleting things. YAGNI, YAGNI. Alongside this, topic-based folders are fine. Not many things are in both "econ" and "sf&f", and when something is relevant to both, such as Krugman's paper on interstellar trade, it's easy to decide where it goes (econ) and where an empty file called "Krugman, Interstellar Trade: see econ" goes. Empty file with a textual reminder, not a softlink or hardlink. Those are brittle. So are attributes. Use broad categories, though. "sf&f", not "SF&F - Fantasy - Urban Noir - West Coast - Mermaids". The definition of "broad" depends on your areas of expertise.

by baz00 3y ago

ThouYS 3y ago

this, tags + dynamic views on those tags

logdahl 3y ago

I'm just wondering when we're gonna get tagged filesystems. Maybe something partially hierarchical. Would love a book/ directory that i can query for 'sci-fi' or 'specification'. I guess one could implement it using sym-links :^)

baz00 3y ago
Don't even go there with the symlinks. My mother made a family tree on windows with folders and shortcuts. It broke OneDrive and robocopy completely. I don't even know how to help her fix it.
- virtue3 3y ago
  It's 3 in the morning and I couldn't stop laughing out loud when I read this. Poor OneDrive lol. I hope she managed to keep all her files and didn't lose anything.
  - baz00 3y ago
    It still works and she hasn't lost anything but you can't actually move or delete any files any more because windows explorer craps out too! I literally have no idea what to do with it :(
    - hobs 3y ago
      My assumption is she ran into the max path length limitation, here's how to fix it https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
      - baz00 3y ago
        That was one of the problems. Unfortunately after fixing that it's still broken.
- alpaca128 3y ago
  On one hand this is an edge case, on the other things like this make me wonder how computers can even work if first-party software built specifically for nothing but managing files fails to gracefully handle a folder structure a user created by hand.
  - baz00 3y ago
    That was exactly my take home from this. Windows has some very serious flaws in it when it comes to data integrity. I do not trust it and do not use it myself.
- mceachen 3y ago
  You could write a short PowerShell script that recursively deletes shortcuts and symlinks, but I'd certainly take a full backup and have it offline before I kicked out off.
theshrike79 3y ago
There are some attempts using Fuse on Linux. At least one I tried ages ago was one you could use to browse your music collection, it automatically generated "directories" based on mp3tags.
Brajeshwar 3y ago
Apple tried doing this with "tag" in the file system but most people don't use it.
- baz00 3y ago
  I use it! :)
  - Brajeshwar 3y ago
    Don’t Do That. Don’t Give Me Hope.
  - smallerdemon 3y ago
    Same.
- hnbad 3y ago
  Microsoft allegedly wanted to get rid of the folder hiearchy completely. There were a lot of rumors surrounding "Windows 97" (what later became Windows 98) because of its Internet integration (which eventually just turned into a lot of dead-end MSN widgets and Internet Explorer being shipped with the OS). I don't know how well-founded this speculation was but allegedly Microsoft wanted to fully commit to things like "cloud storage" (before this was a thing, so of course not using this name) and ended up shelving most of this work because the networks just weren't fast enough at the time. Note: I specifically remember these speculations in the lead-up to the release of Windows 98. They don't seem to be easy to google at this point.
timschmidt 3y ago
May 10, 1997: https://en.wikipedia.org/wiki/Be_File_System https://en.wikipedia.org/wiki/Be_File_System
pedrovhb 3y ago
I just wish xattrs were more resilient and explicit.
Cyphase 3y ago
The author of TMSU left a sibling comment to yours: https://news.ycombinator.com/item?id=37507343 https://news.ycombinator.com/item?id=37507343 https://tmsu.org/ https://tmsu.org/ > TMSU is a tool for tagging your files. It provides a simple command-line tool for applying tags and a virtual filesystem so that you can get a tag-based view of your files from within any other program. > TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever you put them. TMSU maintains its own database and you simply gain an additional view, which you can mount, based upon the tags you set up. The only commitment required is your time and there's absolutely no lock-in.
- notfed 3y ago
  Got me wondering how it tracks file moves. From the FAQ: > TMSU [does] not automatically detect file moves and renames I dunno if we can call it a tagged filesystem with this limitation. (Why not store tags, or at least a GUID, in file metadata?)
  - istjohn 3y ago
    It does store file hashes. It has a repair command that will use file hashes to find moved files. I don't know how well it works.
  - dotancohen 3y ago
    This seems a serious limitation. Bash is my file manager - ls and cd. I would not give that up for any tool no matter how convenient. But presumably there is some API or ABI to monitor, or some signal emitted, when files are moved, added, updated, or deleted. However iwaitnotify works, could that be incorporated (even theoretically) into TMSU?
    - Cyphase 3y ago
      https://github.com/oniony/TMSU/wiki/FAQ#why-does-tmsu-not-automatically-detect-file-moves-and-renames https://github.com/oniony/TMSU/wiki/FAQ#why-does-tmsu-not-au... There are a couple very barebones wrappers around mv and rm, though they could be better (pass through arguments, etc.). https://github.com/oniony/TMSU/wiki/Tricks-and-Tips#filesystem-operations https://github.com/oniony/TMSU/wiki/Tricks-and-Tips#filesyst...
dredmorbius 3y ago
The approach I've been favouring for a while has been to have a data store, I call it the "stacks", based off physical library nomenclature, and then a virtual filesystem which effectively uses searches along various attributes to surface individual items. Those searches might be based on title, author/creator/contributor, publication/creation dates, assigned identifiers, full-text search, relations between documents, subjects or topics, document type / format, or other aspects. How that is specified precisely ... I haven't settled on entirely, though short mnemonics, two- or three-letter where possible (ti -> title, au -> author, date, isbn, oclc, doi, etc.) might work. If the filesystem lives under /docfs, then, say, /docfs/au:barrie/ti:peter+pan would turn up J.M. Barrie's Peter Pan. Specific document identifiers such as ISBNs, OCLCs, DOIs, or LCCSs could pinpoint specific documents, looser searches such as for Peter Pan might generate a list rather than a unitary response but those individual documents would be further distinguished by other properties. The filesystem would be virtual rather than static-on-disk. You're effectively exploring the namespace. Creating this as a filesystem rather than, say, as a database query or application-specific data store means that standard filesystem tools would be available to interact with the contents, though modifications of works might require some additional work. The system could also provide access to works not directly in the stacks, including remote resources (e.g., accessed via URIs and network calls), or applications (through some API). The underlying data store could be in any of several formats, and again the filesystem-based access could abstract between several independent stores. Now all I need to do is build it ...
- Cyphase 3y ago
  This sounds at least somewhat like TMSU, which has been mentioned in some other comments here. https://tmsu.org/ https://tmsu.org/
  - dredmorbius 3y ago
    I've seen that, yes. There are some similarities, and I'm thinking that TMSU might serve as a component of this system.

pbmonster 3y ago

> Tagging/labelling with attributes is the only viable solution that I've found. What's your solution for a new tag being introduced late, which could/should be retroactively applied to already tagged media? Re-tag the entire inventory by hand? Re-tag automatically with some ML/AI tool after you've tagged enough media to train a discriminator? Or just live with old inventory not having that tag?

baz00 3y ago
Depends very much on the problem domain. It's really easy to do this on macOS. So for example I have a top level `Documents/docs` directory. In this everything is recorded in format `yyyy-mm-dd Title.ext` format. I have a saved search which says "give me everything untagged". I will select each thing that is untagged and then tag it!
- plagiarist 3y ago
  I think they're asking something different, what if you think of a brand new tag after you have already tagged things for years? It is potentially a lot of effort to retroactively apply it.
  - smallerdemon 3y ago
    Yeah. All systems like this have to account for ad-hoc additions. Otherwise you will often jam a few squares into some circles just to accommodate your categorization system.
  - cma 3y ago
    AI/LLMs can probably make it easy.
  - dvas 3y ago
    I ran into a similar situation a while back now, I have my version of a zettelkasten system, where I put a tag/ list of tags at the top of the note file (timestamp being the note name). Combining this with something like ripgrep allows me to search the note file contents by those tags, and keywords. A tag I used, didn't seem like the right one for the content as it matured and grew over time. This meant creating a tiny script which would take the name of the tag and the note name, in which it is being used and look at backlinks and forward links and rename the tag based on usage.

imrehg 3y ago

That's what I took away from reading Everything is Miscellaneous a long while back https://en.wikipedia.org/wiki/Everything_Is_Miscellaneous https://en.wikipedia.org/wiki/Everything_Is_Miscellaneous Definitely thinking about categorization differently (folders which are similar to this, tags, organic order by linking; bookmarks, notes, all are in the same boat as well).

lukasgraf 3y ago
I saw David Weinberger's talk at Google [1] some 15 years ago, where he presented the ideas from his book (and bought the book afterwards). This is one of those videos I rewatch regularly, because the ideas hold up so well, even today. His critique of the Dewey Decimal system was the first thing that came to mind when I read Johnny Decimal. And indeed, it is flawed the same way. [1] https://www.youtube.com/watch?v=x3wOhXsjPYM https://www.youtube.com/watch?v=x3wOhXsjPYM

rpastuszak 3y ago

This is precisely the reason why smaller cross-linked notes and tags work quite well for me, and JD or almost anything else in the past hasn't. The ideal for me involves hypertext + (local, private) AI-assisted search. None of this will replace reviewing/pruning your notes regularly (think: Andy Matuschak's fleeting and evergreen notes), but it minimises the friction.

oniony 3y ago

And what led me to build [TMSU](https://tmsu.org/ https://tmsu.org/).

Cyphase 3y ago
Thank you!
vifon 3y ago
TMSU is the only such system I found useful without being cumbersome. After years of trying to use Git Annex, it was refreshing that TMSU doesn't alter the files in any way, merely storing all the (meta)data out-of-band in a separate DB. These days I use TMSU via my own Emacs-based UI almost every single day, so thank you for that!
- dredmorbius 3y ago
  Storing metadata out-of-band strikes me as key to any usable content management system within a realistically complex space. Naming schemes and directory hierarchies have some limited application, but ultimately there will be data which simply won't be shoehorned into any such system, and an externally-managed catalogue tying together disparate elements is required. (Keeping that catalogue up to date and consistent is a whole 'nother issue.) I do like the idea of a virtual filesystem in which elements are effectively search dimensions, which leads to an interesting notion that search is identity. That is, a search will produce one of three possible result sets: - Null, that is, no matches. - Plural, that is, a list of matches. - Unity, that is, one matching item. In the last case, the search providing a single result is an identity of that result. (It may not be a stable identity over time, but it is at least for the present.) Where a list is returned, the size of the list determines how usable it is, and how it is usable. Ten items can be quickly scanned to find the relevant item(s), if they exist. 100 or 1,000 items can often still be managed manually, though they'll typically take some time. Somewhere between 100 and a few thousand items, though, you're in the range where automated assessments or filtering becomes necessary. Large libraries themselves typically have tens of thousands to millions of items. The largest book collections (Library of Congress, British Library) have roughly 150 million books (or equivalents). Other records may exist in greater numbers: periodicals, financial records, databases. Facebook has reported ~5 billion items posted daily for some years now. (I suspect most of those are trivial, but that still leaves a large number of potentially non-trivial items.) Surveillance and other large-scale data collection systems may be larger still.

mihaaly 3y ago

Also if someone by some miracle is able to come up with a univocal hierarchy for a usecase or organization - that may even work throughout time and evolving situation and conditions - then collaborating with an other field or organization will have collisions or discrepancies in taxonomy and a hopeless confusion of category number mapping.

dotancohen 3y ago

I actually use a hierarchy of tags to organize my notes. The _tags_ are hierarchial, not the storage path.

somat 3y ago
but your storage path is already a hierarchy of tags.* *If using a unix derived filesystem.
- dotancohen 3y ago
  I could use symlinks to refer to files, yes, but rather I'm using nonstandard #foo-bar-baz subtag syntax to refer to sections of org mode documents. I do not use real org mode tags because those are restricted to headlines. In other news, if anyone knows how to restrict a search in an org mode buffer to search only the headings, that would be very helpful! Right now I'm using an external grep wrapper, restricting to lines that begin with an asterisk.
  - somat 3y ago
    ehh... symlinks... perhaps, But I am talking about hardlinks. Having said that, I will be the first to admit. the reason hardlinks are not often used is that it is a real "here be dragons" part of the map. 10 years ago I spent a couple fun weeks categorizing all my music. my schema was something along the lines of every file was linked to artist/artist_name/song_name band/band_name/song_name song/song_name/band_name date/YYYY-MM-DD/song_name genre/genre_name/song_name playlist/playlist_name/index etc, etc then I had fun making up a small set of cli utilities to find tags, create and manage playlists etc. then A. it turns out I don't really listen to music. and B. the whole structure wanted to explode every time I would try and move it to a new host. so it probably still sits there on one of my file servers but now as a million duplicated files.

couchand 3y ago

If anything, the history of libraries show you can get really really far by applying a sytematic organizing principle, even if it's WRONG(TM). Librarians may fret that some books are hard to categorize, but library patrons successfully find stacks of books on their own every day all around the world. It's not perfect, but it's better than nothing!

dredmorbius 3y ago
Book classifications ultimately need to assign a book to a location. So long as that goal is achieved, the rationality or consistency of that assignment is a strongly-secondary consideration. If the book has a place, and can be located, where it happens to be relative to other related works, or unrelated ones, really doesn't matter. Searching for works based on similarities or relationships is a problem of the indexing system. I should be able to find works on a similar topic, which reference or are referenced by another, by author, by institution, perhaps by publisher, by publication date, etc. That information would be carried within a catalogue, but need not determine the location of the work within a physical archive.

PurpleRamen 3y ago

> I am convinced that absolutely no hierarchical taxonomy actually works in practice because every categorisation is really slightly fuzzy. Fuzzy is ok, not everything needs to be perfect. > Tagging/labelling with attributes is the only viable solution that I've found. Categories and tags have different use cases. Categories are the best solution, when there is only one possible location for an object, like with physical objects, which cannot appear at multiple locations at the same time. Tags make sense, if you can offer multiple entry points to reach the goal, and have the option to manage them all on a reliable level. And nothing speaks against using them both, with different valuesets. Like with digital files, you can have categories for managing the filetypes (books, videos, etc.), but then use tags to manage the content in each category. > This book is fiction/science-fiction/fantasy. Isn't this kinda redundant? science-fiction is already fiction, it's in the name. Reads more like a hierarchy. Which you of course you can also with tags. Any fantasy-tag automatically inherits the fiction-tag too.

giaour 3y ago
> Isn't this kinda redundant? science-fiction is already fiction, it's in the name. This is nit-picking the specific example rather than engaging with the GP's point. There are definitely genres that span the fiction/non-fiction divide (like romance, horror, politics, judaica, and many more). Interestingly, hierarchical systems used by libraries tend to ignore genre. The Dewey Decimal system doesn't classify literature by genre but instead by language, though some overarching genres (drama vs poetry) are treated as subcategories of language (e.g., German poetry is 831, French poetry is 841), but only for major European languages.
- PurpleRamen 3y ago
  > This is nit-picking the specific example rather than engaging with the GP's point. It's showing that tags can be fuzzy too. So the initial complaint is not avoided, just moved.
  - deleted 3y ago
    [deleted]

eek04_ 3y ago

Scientific libraries categorise into *several categories*. They effectively use hierarchical tags, with the set of tags very carefully curated.

somat 3y ago

tagging is what happens when you allow an artifact to exist under more than one hierarchy at the same time. The progression usually goes. "we want to label our things." "There is a natural hierarchy that we want to be reflected in our labels. "Some things are not labeling well. instead use multiple labels per thing." "There is a natural hierarchy that we want to be reflected in our labels." The unix files system is a good example of the latter. it favors the hierarchy but allows each file to have many names if that is what is required. The main weakness of the unix filesystem is that there is no reverse map, that is, finding a file from its name(s) is simple. finding the names from the file involve searching the whole filesystem.

jjtheblunt 3y ago

I've thought a ton about what you just described, and I think of it this way: not every directed acyclic graph happens to be a tree.

tuatoru 3y ago

> Tagging/labelling with attributes is the only viable solution that I've found. The only solution I've found is being utterly ruthless about deleting things. YAGNI, YAGNI. Alongside this, topic-based folders are fine. Not many things are in both "econ" and "sf&f", and when something is relevant to both, such as Krugman's paper on interstellar trade, it's easy to decide where it goes (econ) and where an empty file called "Krugman, Interstellar Trade: see econ" goes. Empty file with a textual reminder, not a softlink or hardlink. Those are brittle. So are attributes. Use broad categories, though. "sf&f", not "SF&F - Fantasy - Urban Noir - West Coast - Mermaids". The definition of "broad" depends on your areas of expertise.