22 ms·
I am endlessly fascinated with content tagging systems
- Tomte 4y agoAlso great on the topic of tagging, with more information about the AO3 scheme: https://idlewords.com/talks/fan_is_a_tool_using_animal.htm https://idlewords.com/talks/fan_is_a_tool_using_animal.htm
- redbar0n 4y agoA very insightful thread by Hillel Wayne on content tagging systems and their challenges. Their ubiquitous use (in library and information sciences, and popular social networks like Instagram, Twitter, and Pinterest), their deceptive ease of implementation, and "obvious advantages" over hierarchies/folders, means that almost every developer has (or will) run into them at one point or another.. Feel free to comment with good theory and case studies on tagging systems. (It's especially interesting with good case studies for how to model an advanced tag system in a graph database).
- lmkg 4y ago> It's especially interesting with good case studies for how to model an advanced tag system in a graph database I wouldn't accuse it of being a good tag system, nor a true graph database, but one thing to look at is Semantic MediaWiki. It's a MediaWiki extension which takes Categories as a starting point, and extends it quite far with e.g. relations and key-value pairs. One interesting feature of Semantic MediaWiki is called "Concepts" which are essentially "computed tags." They can be used in place of Categories in most places, but while Categories are set by editors on a piece of content, Concepts are defined by a query against Categories or other properties. This can help bridge gaps between different types of tags that represent different ways of thinking about the content.
- blueblob 4y agoA lot of the items described are problems in ontologies
- edflsafoiewq 4y agoYeah. A tag is a predicate. Sub-tags are implication (male author => author). Tag aliases are equivalence (implication in both directions).
- redbar0n 4y agoisn't it super-tags that are implication? A male author (sub-tag) implies it's an author (super-tag). But an author does not necessarily imply it is a male author.
- heliophobicdude 4y agoMy similar issue is with names in source code. Fuzzy matching names and interrogating the contributor about the changes being checked in. Questions to ask the contributor, are the names similar to any of these other names? Is there an opportunity to use the same name or are they different concepts? Code grows and grows and becomes harder to grep if inconsistently naming things.
- micromacrofoot 4y agoThis reminds me of a talk from Clay Shirky about categorization and general ontology. It's interesting to read in hindsight, because it's from when recommendation algorithms were in their infancy. Warning PDF: https://ia800203.us.archive.org/10/items/Ontology_is_Overrated_Categories_Links_and_Tags/shirky.pdf https://ia800203.us.archive.org/10/items/Ontology_is_Overrat... > This is what we're starting to see with del.icio.us, with Flickr, with systems that are allowing for and aggregating tags. The signal benefit of these systems is that they don't recreate the structured, hierarchical categorization so often forced onto us by our physical systems. Instead, we're dealing with a significant break -- by letting users tag URLs and then aggregating those tags, we're going to be able to build alternate organizational systems, systems that, like the Web itself, do a better job of letting individuals create value for one another, often without realizing it.
- dadadad100 4y agoThank you for this link. I’ve been looking for a good discussion of the browse vs search argument and this is very, very good
- UltraViolence 4y ago
- cptcobalt 4y agoI can't wait for the author of this thread to discover the AO3 tagging system, which is, frankly, a masterpiece that demonstrates how effective community management can lead to extremely good tagging and categorization, with very little miscategorization. https://www.wired.com/story/archive-of-our-own-fans-better-than-tech-organizing-information/ https://www.wired.com/story/archive-of-our-own-fans-better-t... https://archiveofourown.org/faq/tags https://archiveofourown.org/faq/tags
- rubinlinux 4y ago> The only system I know that does that is the fanfiction site AO3, where teams of volunteers manually create aliases from, say, "snarry" to "Harry/Snape" They seem aware already.
- swyx 4y agoits literally the third tweet in his thread
- at_a_remove 4y agoThe AO3 tagging system badly needs pruning. I hesitate to make examples, as the specificity will serve as a "call out," but quite a lot of authors throw in single-use, digressive tags as some kind of commentary on their own work. Huge meandering swaths of crap tags, and the people who make them ought to have their permissions to create tags revoked.
- PuppyTailWags 4y agoI kind of disagree with this. Tags are dual use in AO3, specifically they serve as a way to find specific stories with specific thematic or plot elements, but they additionally serve as a free expression of the author because its the author who chooses which if any tags they want to use to describe their piece. When an author gets to decide the categories of a work, the categorization also becomes an expression. Consider the flavor of "Dead Dove: Do Not Eat" tag, which serves both as an author's expression of warning the reader and also a category of fanfic that is expected to have transgressive elements. Just tagging, idk, "child endangerment" completely misses the point of "Dead Dove: Do Not Eat" comparatively.
- kortex 4y agoI think tag aliases are fine, but in my opinion, tags should not have hierarchies. That is just opening the can of ontology worms, and most systems are ill-equipped to deal with ontologies...including ontological systems. Tags are just dumb strings which label data. They are basically KeyValues, where the value is just always equal to True. We don't think of KVs as hierarchical unless they are explicitly a path string, and in that case, they are forced to be a plain tree with no cycles or diamonds.
- comfypotato 4y agoOrg mode approaches this by making hierarchies and inheritance optional. I personally like both, but I acknowledge (as was mentioned in the tweets) that hierarchies can get to be very convoluted if you don't work to maintain them sensibly.
- AlanYx 4y agoWhat I like most about org mode tags is that regular expressions can be subtags (or "members of a group tag" in org mode lingo). So you can specify a hierarchy where the parents have children you don't know in advance.
- zamubafoo 4y agoOptional forests of hierarchy trees are where it's at. Essentially don't encode everything into one gigantic one. Sometimes you know that users are going to tag `laptop` a bunch and want that to also drag in `personal computer` (but not all `PC`s are `laptop`s) or that `blue dress` is also a `dress` and don't want to hard code special cases. That said, if you are going to do this, then you must have it controlled by an admin/moderator. Maybe allow for hierarchy request submissions but have it moderated. There is at least one public system where this just works to my knowledge and a bunch of self-hosted ones as well.
- goto11 4y ago> They are basically KeyValues, where the value is just always equal to True That would be a set of values :-)
- counttheforks 4y agoAnyone have a suggestion for a tagging filesystem that is maintained? Or if not a filesystem, something that at least works? I still feel like this is the best way to organize personal photos and media, and while https://www.tagsistant.net/ https://www.tagsistant.net/ is pretty good it hasn't been updated in 6 years and is fairly buggy.
- comfypotato 4y agoDr. Karl Voit did his dissertation designing a tag-based file system. I don't know what the status is today, but the dissertation itself may be a decent place to start your search. https://karl-voit.at/tagstore/en/papers.shtml https://karl-voit.at/tagstore/en/papers.shtml
- btrettel 4y agoI haven't tried it yet but https://tmsu.org/ https://tmsu.org/ is actively maintained and looks nice.
- greggman3 4y agoMacOS has tags. Right click any file in finder, select "Tags..." No idea if they are implemented at a filesystem level but there are various tools for finding things by tag
- somat 4y agounix has tags, they are known as hardlinks.
- iggldiggl 4y agoYou can misuse them as such (I've done it myself, albeit on Windows), but they're not really made-to-measure for that use case. E.g. deleting files that have been "tagged" that way becomes more cumbersome (because to the OS all hardlinks are created equal, so deleting the "primary" file doesn't automatically delete the tags, too), it's incompatible with any program that uses "atomic" files saves instead of modifying files in-place, the UI for viewing and editing "tags" is not really there, …
- 4y ago
- jrochkind1 4y ago> I can't find anything on how to design and implement anymore more than the barebones basics of a system. All of this stuff (horse/horses etc) is extensively discussed, maybe look under "taxonomy" or "ontology". Now, whether you want to use any of those solutions or not or find the discussion useful or not... if you aren't finding anything about it at all, you aren't looking in the right places. (I learned about it in librarian school)
- edflsafoiewq 4y agoCan you link some resources about it then?
- tantalor 4y agohttps://en.wikipedia.org/wiki/Library_and_information_science https://en.wikipedia.org/wiki/Library_and_information_scienc... https://en.wikipedia.org/wiki/Tag_(metadata) https://en.wikipedia.org/wiki/Tag_(metadata)
- jrochkind1 4y agoI could, but honestly I'd just be googling "taxonomy". But ok that's not entirely true, I know how to refine my search and recognize when something is what I'm thinking of, from some familiarity with the field. (But if you want to look around, in addition to "taxonomy" and "ontology", other good terms are "information architecture" and "controlled vocabulary"). These are not things I have vetted, this is literally just me googling and taking a quick skim... https://blog.optimalworkshop.com/how-to-develop-a-taxonomy-for-your-information-architecture/ https://blog.optimalworkshop.com/how-to-develop-a-taxonomy-f... https://www.uxbooth.com/articles/introduction-to-taxonomies/ https://www.uxbooth.com/articles/introduction-to-taxonomies/ https://www.nngroup.com/articles/taxonomy-101/ https://www.nngroup.com/articles/taxonomy-101/ http://accidental-taxonomist.blogspot.com/2020/11/what-it-thesaurus-and-what-is-it-good.html http://accidental-taxonomist.blogspot.com/2020/11/what-it-th... Or how about some textbooks: https://narrowgaugebooks.indielite.org/book/9781627055802 https://narrowgaugebooks.indielite.org/book/9781627055802 https://www.hedden-information.com/accidental-taxonomist/ https://www.hedden-information.com/accidental-taxonomist/
- tra3 4y agoI've been dabbling in personal knowledge bases for a long time now. I remember the when I discovered tags -- thought it was the best thing ever. The first good implementation in the wild (for me) was del.icio.us. Eventually I ran into all the problems that the linked thread describes. "Movie" or "movies"? "Book" or "books"? In any case, I still think flat tag lists are better than a directory tree structure ("Content/Movies" vs "movies, movie, entertainment, science fiction, space travel, aliens"). A recent innovation that I'm enjoying is backlinks. I believe roam research was the first major player that showed you related entries via the links that you included, even though a similar concept existed forever. Then you can generate clouds of relationships and find concepts visually [0]. 0: https://noduslabs.com/cases/visualize-connections-notes-roam-research-infranodus/ https://noduslabs.com/cases/visualize-connections-notes-roam...
- jazzyjackson 4y ago> backlinks > recent innovation Ted Nelson is rolling in his Xanadu
- tra3 4y ago100% there was prior art to this, I was thinking zettelkasten. Didn't know about Xanadu though!
- PeterStuer 4y agoLook into AI systems from the 1960's and you will find Semantic Networks. If you just need categories you can go with taxonomies and folksonomies. If you want to (over?) formalize and describe mainly non-agentive structure you look at ontologies.
- labrador 4y agoI am too but I've given up. I've collected a lot data over the years and spent a lot of time trying to organize it so I can find relevant connections. It's just too time consuming. I've decided discerning relationships in unstructured data is where I want to focus.
- raffraffraff 4y agoI'm largely the same with the exception of music. I don't use online music services, and have had the same (growing) collection of mp3/m4a/flac files since the late 90s. I have a few custom tag fields that I maintain in an ad-hoc fashion. First, I'll add new music to an inbox and give it a few listens. If anything jumps out at me I'll give it a rating, and maybe add some tags under 'mood' and 'instruments'. And that's about it. It doesn't have to be perfect and it doesn't require a lot of time. Maybe I'll spend 30 minutes a week going through my recent play history tagging specific tracks that I like. It's nice to be able to put on "uplifting saxophone" or "dark dramatic piano"
- cpsns 4y agoI’ve written a tagging system from scratch for an existing system and it was one of the most interesting things I’ve worked out. I had total control over how it was implemented and I think I came up with a really nice, minimalist, scalable way to tag things, and to search them.
- rambambram 4y agoCare to elaborate? I'm also working on a categorization/tagging system - albeit a simple one - and I find myself in a struggle to keep it accessible enough to use on one hand and advanced enough to actually add value on the other hand.
- redbar0n 4y agoIf you share a brief overview of it, you might receive some good critique of it here (surfacing potential pitfalls others have run into with similar systems). Besides, it might be helpful to others.
- aaws11 4y agohttps://threadreaderapp.com/thread/1534301374166474752.html https://threadreaderapp.com/thread/1534301374166474752.html
- xcskier56 4y agoThe hierarchical nature of the information he's talking about really reminds me of the ontologies and terminologies that are used in healthcare to organize medical information. E.g. Ibuprofen 10mg Tab < Ibuprofen < NSAID < ... < Therapeutic Chemical. This is a field that I'm only tertiary familiar with but it's a fascinating discipline trying to group, and manage all of the different categories of healthcare data. You can use the RxNav tool to look at the RxNorm terminology which is only 1 of many terminology systems. https://mor.nlm.nih.gov/RxNav/search?searchBy=String&searchTerm=ibuprofen https://mor.nlm.nih.gov/RxNav/search?searchBy=String&searchT...
- swyx 4y agowe have a big tagging problem where i work and yesterday I tried using gpt3 to assist. worked well! code and context: https://github.com/airbytehq/airbyte/issues/17893 https://github.com/airbytehq/airbyte/issues/17893
- VectorLock 4y agoOne example of an unexpectedly rich and deep tagging ontology is the Danbooru "Anime" image board [NSFW] https://danbooru.donmai.us/ https://danbooru.donmai.us/
- system2 4y agoI know this is not reddit. But why do you know even know this link and its tagging system...
- VectorLock 4y agoI'm not scared away by things that might offend the puritanically inclined and I'm interested in ontologies and this is a fascinating one. There was some drama about someone training a Stable-Diffusion-alike by ripping their dataset that brought it to my attention.
- thrdbndndn 4y agoDanbooru is one of the most popular anime image board. Anyone who's into Anime (not just for hentai) probably knows.
- TazeTSchnitzel 4y agoThere is a safe-for-work, or at least safer-for-work version of the site: https://safebooru.donmai.us/ https://safebooru.donmai.us/ (It is of course based on the tagging system: every post is tagged by its "safeness" level.)
- thrdbndndn 4y agoYeah, danbooru or similar image boards basically have all the things talked in this tweet thread. They have tag aliases, meta-tags and so-called "tag implications". The last one is basically sub-tags but with more flexibility and dead simple to implement: if A implicates B, then tagging an image with A will automatically tag it with B. So you can tag "American Male Novelist", and then the system will automatically add "American", "Male", "Novelist", "Writer", etc. (after such implications were added). It much easier than Wikipedia's categories, but Wikipedia's way is of course intentional because categories is meant to have a stronger hierarchy than mere tags.
- at_a_remove 4y agoThis seems like one of those Eternal Problems that people, whether librarians, programmers, or hobbyists, stumble across, think they'll make headway in, then discover that they've really managed to progress just a few feet across a vast and hostile surface of landmines, pitfalls, and lures. Each "obvious" step (I'll have parent relations to define a context!) is only yet another bargain with the Devil, who laughs at your precautions.
- MisterBastahrd 4y agoI guess if you're really focused on it. I built a content tagging system for an old employer that would attempt to guess context based on keywords and associations but give the writer of the content the final say in what's actually being tagged. Sure, I could have spent a thousand hours refining it, but the improvement would have been marginal and it still would need human interaction.
- csours 4y agoWas it used for content related to that particular business? I think as long as you have relatively limited variety, you can make something that works well enough.
- at_a_remove 4y agoSimilarly, I think if you have a limited number of people doing the classification, you can also make a good shot at it.
- MisterBastahrd 4y agoThe content was a publisher. Writers would submit articles, the system would automatically tag them, and the system was good enough that usually the writer or editor would weed out a false positive or two, which is about the same result as all these machine learning use cases with thousands of hours of dev time. IIRC, terms were weighted, so that some of them needed to have more instances in the articles than others in order to be included in the final tag results. Locations were one-offs, but specific topical items required more mentions because of false positives. And then there were things we called branch-offs. Branch-off tags occurred when a topic was mentioned enough to be a tag but there's another name that some segment of the population would know it by. For example, the fish known as the white crappie are known in Louisiana as sac-au-lait, but people would also spell the word sacalait or sac-a-lait. So when we would get an article from an author in the Carolinas, they have no familiarity with a term that is the dominant one in Louisiana, but the software would add the tag anyway, which also exposed it to our site search.
- aaviator42 4y agoA few months ago I worked on some proof-of-concept code for searching tagged data: https://github.com/aaviator42/Cha https://github.com/aaviator42/Cha I now work full-time in a role where part of my duties is designing a content tagging system and its search functionalities. It's very interesting and fun! Lots of puzzles. How do you weigh different tags? How do you do fuzzy searching ('city' should match with plural ('cities'), misspellings ('citys'), etc)? How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? What about synonyms? What about regional terminology and synonym tables? Then there's one-to-one and one-to-many and many-to-one mapping. As a side project I'm also working on a beta public search engine that I'll launch on HN sometime in the next year or so, where I'm having similar puzzles.
- cube2222 4y ago> How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? That sounds like a very good use case for word embeddings.
- bell-cot 4y agoHow do you deal with "hotdog" possibly being a noun (several meanings), or proper noun (several meanings), or verb, or interjection?
- dmonitor 4y agoe621 frequently has to deal with characters with the same name, or an artist with the same name as a character. they just make ambiguous tags have a special syntax. so if bob was an artist, but also had a character named bob, it would just be bob_(bob) for the character and bob_(artist) for the artist. and if someone tried to tag something as just “bob” they would be told to be more specific. searching for all bobs can be done with bob_(*). so hotdog could have hotdog_(food), hotdog_(interjection), and hot dogs (the animal) would be two tags: hot and dog. it’s not the cleanest solution, but it works well enough.
- papito 4y ago
- wtf77 4y agoI am endlessly fascinated by how twitter has now become a dumping ground for complex topics that are difficult to read and follow. But what happened to the old blogs?
- dymk 4y agoIt's lower effort to make a stream of consciousness post one sentence at a time, and as a bonus, there's a built in audience / discovery network where they're posting.
- tuatoru 4y agoLower effort for whom? Back when I were a lad, we were told to write so that our readers did not have to work to understand us. The point of writing is to be understood. Old man yells at cloud.
- dymk 4y agoLower effort to the writer, obviously. The point of posting on Twitter is not to be understood, it's to be retweeted.
- tqi 4y agoI think it's helpful to keep in mind that with most of examples that get shared around, the choice for the author was not a string of tweets vs blog post, but rather a string of tweets vs not sharing at all.
- dylan604 4y agoHow does one go back and edit a stream of consciousness like that into an actual coherent thought later though? I was just having a conversation similar to this where it was explained "this is just how people my age do things". While attempting to avoid boomer/millennial tropes, this does make me wonder how much different schooling is now vs then (hoping to avoid those memes too). I was always getting in trouble for just saying whatever came to mind vs slowing down to think if it really needed to be said or more specifically how it was said.
- contextfree 4y ago"Advice: don't let the tag predicates refer to other tags" But then how would I search by the tag of all tags that do not tag themselves???
- joshu 4y agothere's a massive difference between tagging-for-self-recall and tagging-for-other-recall. when i invented tagging the first was paramount, but the latter has become dominant and has very different design considerations one interesting note: you can infer a bunch of hierarchical information since people frequently tag from broader to more specific, topicwise. some things can be tagged by multiple people and you can thus infer synonyms as well. this can thus be fixed in search.
- milesskorpen 4y ago"When I invited tagging" is such a flex. But creating delicious gives you some credible claims there.
- joshu 4y agoI don't get to use it much these days
- holub008 4y agoContent tagging in online systems has existed since at least the 1970s, with the earliest example I can think of being MEDLINE https://en.wikipedia.org/wiki/MEDLINE https://en.wikipedia.org/wiki/MEDLINE
- joshu 4y agometadata is not is tagging. keywords are not tagging.
- holub008 4y agoRight, MEDLINE's "tagging" system is MeSH, which is a large controlled vocabulary. MEDLINE does contain bibliographic data + journal keywords, but its real value add is MeSH, which is used for search, related publication identification, etc. in PubMed. https://en.wikipedia.org/wiki/Medical_Subject_Headings https://en.wikipedia.org/wiki/Medical_Subject_Headings
- NWoodsman 4y agoIn my app, users apply a set of tags to a note, but then the app automatically creates hierarchical associations in a tree. There are an exponential number of associations between tags (At one point design was failing because it was trying to prebuild 100k+ GUI items for these cross-referenced tags) so I had to virtualize the intersection of tags at the exact moment a user expands a tree item. You cannot plan what tag search will lead you back to the data you want, so every node in the graph must be bidirectional.
- philip1209 4y agoWe spent a lot of time building tagging systems to organize technology skills on https://www.moonlightwork.com https://www.moonlightwork.com. The coolest part was training a collaborative filter on the tags. So, when you add "Django" as a skill, it could recommend "Python" as a related skill. This made for some refined user experiences. Getting typeahead search right took a lot of refinement. Here is some of the logic we ended up implementing over time: 1. Exact matches get prioritized first (e.g. "Go") 2. Abbreviations support (e.g., "AWS" for "Amazon web services" or "ROR" for "Ruby on Rails") 3. Name that start with query should go before non-leading matches (e.g., "Ru" should return "ruby" before "task runner") 4. We tracked an "Aliases" column for each tag to enhance search. So, "golang" was an alias for "go".
- somat 4y agoMy (Chomskyish)hierarchy of tag systems goes something like. tagged data key=value tagged data hierarchically tagged data (we just found the the unix filesystem!) hierarchical key = value tagged data (oh damn, it's ldap, we dug too deep.)
- deleted 4y ago[deleted]
- terpimost 4y agoI was interested in that too. I stopped when as soon as I realized that any good search in tagging system would be just a full text search. E-commerce catalogs have detailed filters but I think people use maximum 2 properties in addition to simple name input search
- photochemsyn 4y agoOne area that's illuminating is the effort to annotate the results of whole-genome sequencing projects. Tagging stretches of the genome which represent coherent units of some sort, and then relating them to some functional capability of the organism, is not at all a solved problem. Here's an overview from 2011 where they're struggling to even get a good tagging system up for single-celled microorganisms (a much easier problem than multicellar genomes like humans): https://pubmed.ncbi.nlm.nih.gov/22180819/ https://pubmed.ncbi.nlm.nih.gov/22180819/ > "Highlights include the development of annotation assessment tools, community acceptance of protein naming standards, comparison of annotation resources to provide consistent annotation, and improved tracking of the evidence used to generate a particular annotation. The development of a set of minimal standards, including the requirement for annotated complete prokaryotic genomes to contain a full set of ribosomal RNAs, transfer RNAs, and proteins encoding core conserved functions, is an historic milestone."
- pessimizer 4y ago> It gets even more complex if tags can have multiple parents, like Wikipedia categories. "American Male Novelists" is a subtag of "American Male Writers" and "American Novelists". Now we have diamond problems, redundancy, a whole host of other edge cases. I don't understand this problem. I would think that you would have tag:american tag:male tag:novelist tag:writer, and tag:novelist would itself be tagged as tag:writer, because all novelists are writers.
- dahdum 4y agoI adore tagging systems and have worked on them in several different applications and implementations, but there are always pitfalls and trade offs, and it’s possible to bury yourself Nowadays I nearly always store the assigned tags as an integer array column in Postgres, then use the intarray extension to handle the arbitrary boolean expression searches like “((1|2)&(3)&(!5))”. I still have a tags table that stores all the metadata / hierarchy / rules, but for performance I don’t use a join table. This has solved most of my problems. Supertags just expand to OR statements when I generate the expression. Performance has been excellent even with large tables thanks to pg indexing.
- justinpage 4y agoWould you mind sharing a simple example that demonstrates this? Sounds great!
- srcreigh 4y agoDo you index arrays? What index type is that? Any tips? I’ve used array column in PG before, haven’t indexed arrays though.
- marcosdumay 4y agoAFAIK, postgres first got its reputation of high performance because of array indexes. People usually go with GIN indexes, that can be used on the contains, overlaps or equals comparisons.
- dahdum 4y agoYes, it’s explained in the intarray doc here. GiST is the one I use, but as it states GIN should be faster on reads. I haven’t really thought about that in many years, I should run some perf tests. https://www.postgresql.org/docs/9.1/intarray.html https://www.postgresql.org/docs/9.1/intarray.html
- RussianCow 4y agoThe tradeoff here is that you lose the foreign key constraint, correct? So if you delete a tag, there is no way for the database to automatically remove all references to it. Or is there some way to do this now?
- endisneigh 4y agoIs there an optimal tagging system, performance wise? Seems like there could be a database just for tagging.
- redbar0n 4y agoProbably a graph database. Considering graph dbs are optimised for JOINs (doesn't need to do them, due to direct relationships between individual records aka. index-free adjacency). The question is how you would effectively model the tag system in a graph db, as there are several ways to do it.
- jshandling 4y agoI set out building my first full-stack webapp [0] to make a custom theme-based tagging/organizational system for musical ideas. I did not initially realize all the hairy design choices inherent in this domain, but have found it humbling and educational. Remaining features to be implemented include in-app audio recording, editing, and custom labeling outside of the main tree structured organizational system. I'd appreciate any thoughts or suggestions if anyone cares to take a look! [0] https://www.soundseeker.app/ https://www.soundseeker.app/
- turnsout 4y agoThis is the reason the Semantic Web never took off—people on the internet can't even agree on what a "sandwich" is, let alone the exact hierarchy of ontology. This is an area where large language models have a role to play—whatever you're hoping to achieve with user-generated tags can probably be achieved with ML-powered associations or navigation. And the potential benefit is that it could be tailored to each user—so you're only surfacing "Hot Dogs" when certain users click "Sandwich."
- CabSauce 4y agoThis is what we do for the most part. Two tiers of 'tags'. One is curated and required, the other is an embedding.
- CobrastanJorji 4y agoI thought that the cube rule of food generally settled the sandwich debate. A hot dog is not a type of sandwich, being surrounded on three sides. Instead, it is a type of taco.
- turnsout 4y agoHaha yes, and I would hate to meet the Turing-complete tagging system that could capture this nuance!
- subpar 4y agoI've done this professionally in a couple different settings, from building topic classifiers for news events (it is sometimes hard to know when one news event should stop and another start) to creating tagging systems for audio recordings of group conversations (where topics often merge in and out of each other, often within a single sentence). I'm currently working on classifying non-speech, non-musical sound and it can be useful to piggyback on an existing knowledge system, though they tend to be industry-specific. As an example, Google's ontology for sound identification [1] is a nice starting point for general classification, whereas the taxonomy [2] used by the audio post-production industry (sound effects, foley, etc) is structurally quite different (which isn't surprising, but it sure is fun!). From a totally different field (electro-acoustic composition), the work of Michel Chion and Pierre Schaeffer [3] add psychoacoustic elements to more traditional measurable characteristics, i.e. how the sound is perceived and comprehended is just as important as its medium of travel and its source. It is helpful to see what others have done before you so you can pick and choose elements of their work to incorporate into your own. 1: https://github.com/audioset/ontology https://github.com/audioset/ontology 2: https://docs.google.com/spreadsheets/d/1b2UhKpcOAE-jd1edOsxCJALqttyL761h/edit#gid=1490464926 https://docs.google.com/spreadsheets/d/1b2UhKpcOAE-jd1edOsxC... 3: [big pdf!] https://monoskop.org/images/0/01/Chion_Michel_Guide_To_Sound_Objects_Pierre_Schaeffer_and_Musical_Research.pdf https://monoskop.org/images/0/01/Chion_Michel_Guide_To_Sound...
- robocat 4y agoThe Google list is interesting. It only has main category and subcategory. Causes problems like “SCIFI WEAPON SCIWeap = Lightsabers, exotic sci-fi weapons. Not a 'blaster' which would go in LASERS-GUN.” because they created another main category LASERS for the desired four extra sub-subcategories (hmmmm, how should that be spelled? Categorically another conundrum.)
- openfuture 4y agoWhy twitter man.. these questions are clearly important but there is a space to discuss them https://matrix.to/#/#datalisp:matrix.org https://matrix.to/#/#datalisp:matrix.org
- googlryas 4y agoI had to click like 5 links from that link in order to get to a site which requires me to sign in before allowing me to see the content. I still have no idea what I'm supposed to be seeing. And no idea what the connection between "datalisp" and content tagging systems is. Maybe that's why twitter man?
- openfuture 4y agoYeah I am not a fan of the login walls and all that either but there is a reason that we should try to use free and open source software and currently matrix is the option that is convincing enough to use. Datalisp.is the web of trust / semantic web / whatever. It doesn't exist outside my head currently but it also exists in lots of other heads (at least bits and pieces) so I believe we should manifest it.
- hoherd 4y agoSeriously. I'm not a twitter fan, but even so, it's a short-form medium. Why do people abuse it like this, especially with great content? What's so bad about tweeting a link to a blog? Anyhow, I use threadereaderapp to get through the frustrating twitter UI and the ways that it is abused: https://threadreaderapp.com/thread/1534301374166474752.html https://threadreaderapp.com/thread/1534301374166474752.html
- modriano 4y ago> Why do people abuse it like this, especially with great content? Probably to get a wider audience to actually read and engage with the ideas, and to crowdsource relevant information from said audience. > What's so bad about tweeting a link to a blog? Probably an 80%+ reduction (total guess) in the number of people who engage directly with the content and author.
- xg15 4y agoI worked with the Wikipedia category system a few years ago, and you could see the problems with hierarchical tagging systems right in action back then. (Though it may have gotten better in the meantime) The system appeared simple: There were just two relations, "Article A is a member of category B" and "Category X is a subcategory of category Y". However, in practice, the community was using this system to represent a whole host of wildly different relationships between items, often with different implications what a category actually applied to. E.g., if A has a subcategory B, this could mean one of several things: B might be an additional constraint on the items in A ("American writers" -> "19th century American writers"), the things in B might be more specific than the things in A: ("Writers" -> "Novelists"), A might apply to the concept B, not the things in B ("Occupations" -> "Writers") or A might refer to the category B ("Categories with more than 100 entries" -> "Writers") and on and on... Of course those different aspects could even be combined. E.g. "Categories with more than 100 entries" might have a child "Categories with more than 100 entries in need of review", which represents a constraint but might itself contain less than 100 entries... The basic question "Is item X in category Y" becomes impossible to answer generally, because there is no clear indication if a category only applies to its direct children or to all of its descendants or only to the subcategories itself. I'm sure there are sophisticated ontological systems which would allow users to specify all those different relationships separately. I'm also pretty sure that users would become sloppy after a short time or would disagree which particular relationship to use in a particular situation...
- crazygringo 4y agoI encountered the same problem a few years ago and indeed realized that using categories to understand what type of article a thing was (person? subject? event?) was utterly useless, for the reasons you describe. On the other hand, I discovered that infoboxes (the data in the top-right box on most pages) was generally extremely reliable, if frustrating to parse.
- maxbond 4y agoThe infoboxes are created from a query to Wikidata, which you can query yourself! No scraping necessary! https://query.wikidata.org/ https://query.wikidata.org/ You'll want to learn SPARQL, but if you know SQL it's not so bad to pick up.
- deleted 4y ago[deleted]
- k__ 4y agoI don't know much about this topic. The only thing I learned: if you think you have a taxonomy, then you don't.
- robg 4y agoSurprised no one has nailed a use case for semantic tags and their associations. Python and snake doesn’t require hierarchies to differentiate from Python and coding. Why aren’t co-occurrences within and between content samples enough?
- PaulHoule 4y agoPeople look forward to a visit with the ontologist they way they do a visit with the orthodontist.
- taylorbuley 4y agoPro tip: use stemming!
- throwaway920102 4y agoEmpornium aka luminance has a great tagging system.
- dekervin 4y agoI hacked together a small extension to tag hacker news stories. A small presentation here, https://datum.alwaysdata.net/static/extension/index.html https://datum.alwaysdata.net/static/extension/index.html With the js files for the extension. The motivation to finish it partly came from this hn thread. https://news.ycombinator.com/item?id=32970560 https://news.ycombinator.com/item?id=32970560
- didip 4y agoIf you don’t want to think too hard, just funnel the tags information into a search engine like Elastic Search. It already handles stemming, stop words, aliases, etc.
- errantmind 4y agoInstagram's tagging system was actually really effective at categorizing content and discovery because each hashtag was treated as a node in a (giant) graph, where each node has multiple properties, including post count (number of posts using a tag), 'velocity' (number of posts using a particular tag per unit time), etc. I could write up a big post about it as I made a study of it in when I created a web app for finding the most relevant tags a few years ago. All that to say there was a lot to their system and it worked because users became aware that they were rewarded for using the most relevant tags. Using irrelevant tags was punished. This guided users towards using a mix of relevant popular and niche tags to maximize their reach, which, in turn, further improved the tagging system. Instagram's tagging system isn't as important anymore as their algorithm has deemphasized it, in favor of other methods for classification and discovery, but there were a couple of golden years where it worked very well. Most users still look back on those years as the 'good times' even if they don't know exactly why. I'd go so far as to say they ruined the app after they deemphasized tags (and added way too many ads)
- emj 4y agoOpenstreetmap is map data that is basically coordinates with tags on them and relations between those tags. I guess this is true for most GIS software but there is very little 2D map data that can not be described in the OSM tagging model. You can never express everything with tags, you need stats and metadata on metadata, documentation and a strong heterogeneity which also need to be able to adapt to new ideas. https://wiki.openstreetmap.org/wiki/Tags https://wiki.openstreetmap.org/wiki/Tags https://wiki.openstreetmap.org/wiki/Map_features https://wiki.openstreetmap.org/wiki/Map_features
- matkoniecz 4y agohttps://wiki.openstreetmap.org/wiki/Tagging_mailing_list https://wiki.openstreetmap.org/wiki/Tagging_mailing_list ( https://lists.openstreetmap.org/pipermail/tagging/ https://lists.openstreetmap.org/pipermail/tagging/ ) is a fascinating, hilarious and interesting place. Basically it is about an endless attempt to classify at least part of reality, in organically growing worldwide project based on bunch of passionate obsessive hobbyists with overly strong opinions. With bonus of bunch of politics, confusion and passion. https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_API_by_Example https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_AP... is likely of interest.
- roberthahn 4y agoI'm so happy to see people talk about this! I too am endlessly fascinated with content tagging systems. Hillel's thoughts are completely unsurprising to me so I guess I've come to similar conclusions. I do notice that we seem to care about different things though - where Hillel appears to focus on tag types (and the implementation challenges that go with that) I focus more on human factors like what problem are we solving? for who? How do we maintain relevance (and power) in tagging systems (and for who?) I'm of the opinion that tagging systems should not be made by the few for the many but by each person for themselves. Which, of course, sucks because that puts the onus on everyone who wants tagged content to do their own work. But I believe the output of that investment would be quite valuable and useful! An easy example I could use might be recommendation engines. Assume I have a database of tags (a tag cloud?), and I know you have similar interests to me. If you also have a tag cloud, I could input links to both of our tag clouds into a purpose-built recommendation engine to discover new content I might not have consumed yet.
- jsemrau 4y ago> I could use might be recommendation engines. Assume I have a database of tags (a tag cloud?), and I know you have similar interests to me. If you also have a tag cloud This was the first "naive" implementation on finclout. Every post get automatically scanned for ranked keywords and then matched with other known entities about the post. We also user collect tags from the user and have users verify keyword matches.
- redbar0n 4y agoWhat made you move away from that "naive" implemeentation? What kind of implementation do you now employ?
- ggm 4y agoApproximate date is the bugbear of photo tagging. EXIF and Dublin core and vendors can't agree what to do. Camera manufacturers don't care because at time of shot, date is fixed. It's archival, scanned and copied predigital work.
- mceachen 4y agoFor what it's worth, ExifTool (and by extension, PhotoStructure) support 0 for month and day. The problem is that most all other applications won't see this as a valid date. And I've struggled with how to covert this "fuzzy" date into something that sorts with other assets that _do_ have an exact date. Should they all live on the first of the month? In the middle of the fuzzy date range? Midnight? Noon?
- ggm 4y agoYes, I like your software and paid for it. But as you note it's not a solution guaranteed to be portable to another photo framework or even Microsoft file tag management. DC discussed approximate dates at some point. What does 'circa' 1900 mean to searches by date? Is 1892 circa 1900. It's hard.
- Archelaos 4y agoThose interested in the state of the art of professional tagging systems in culture heritage may have a look into the CIDOC Conceptual Reference Model (CRM): https://www.cidoc-crm.org/ https://www.cidoc-crm.org/
- josefrichter 4y agoI was fascinated by ontologies 10 years ago. Since then, I've been studying human brain, only to realize that this is an effort to basically build a software version of human brain. Maybe it's possible, but it's definitely not feasible in 99.9% of cases. The closest thing we have is some machine learning approaches.
- system2 4y agoI am increasingly hating twitter being used for blogging.
- ok_dad 4y agoI like how they worked out an advanced tagging system's requirements from a ~dozen tweets, starting with the most basic tagging system and working up through a tag hierarchy to a tree to a DAG, then even talks about K/V tags and etc.
- polote 4y agoA big miss on the list, is that words (so a tag) do not mean the same things for each people and do not even mean the same things in different contexts
- PaulHoule 4y agoMaybe it's the project I am working on but right now I see the ideal search interface to be something like an OWL class axiom, that is, I am searching for instances of a class that has the following restrictions * subclass of Actor * subclass of Singer * has been in at least 7 movies * was born after December 3, 1980 * has been married to at most 3 other people these can be intersected, unioned, complemented, etc.
- somat 4y agoIt sounds like what you want is SQL. There is no good solution for the cultural problem that a written language is somehow unsuitable for end users. but personally I have spent way too many hours trying to make a search interface only to realize at the end that not only is my interface complicated and hard to use it still has only a fraction of the descriptive power a sql query has. At times I am tempted to make full use of the built in database permissions and let the user just type queries directly. but this suggestion is always vetoed.
- AtlasBarfed 4y agoEh, the diamond problem and transitive issues don't exist because what is being reduced to is simply a set and membership. if expansions / aliases / synonyms / multi-membership produce overlaps, who cares, it's a set of hashs. The overwrites only represent wasted computation. Really this is a simpler version of multiple inheritance. You don't have the issue of conflicting method signatures and implementations, only names. The only danger is names meaning different things. You need your tags to be relatively unique to the meaning.
- aaron695 4y ago
- pphysch 4y agoMy current solution to this problem is just putting a JSONB column in relevant tables. GIN indexes do the heavy lifting as needed. This lets us implement arbitrary, queryable ontologies on top of the data without requiring further database instrumentation (aside from creating an index now and then).
- qwerty456127 4y agoThis is crazily sad non-invasive (without embedding into the file body) tagging is not standardized across OSes and file systems. The only system to support tags I know is KDE/Dolphin/Baloo, outside KDE tagging seemingly is supported only by a handful of incompatible 3-rd party apps. Sadly I don't expect much progress to happen in this area. Almost nobody cares about storing and organizing of files locally nowadays. I hope it is going to be done some day or later (there isn't much to do: just standardize some xattrs and something like RDF schema to be used in an alternative FS stream + add support for these to the standard file management and search tools, this is orders of magnitude easier than implementing a new FS) but probably not soon - it would be a huge luck to get any resources allocated to this.
- raffraffraff 4y agoI'd love to know what those prolific Spotify engineers think of this. That was a joke because Spotify doesn't let you tag music.
- sirspacey 4y ago(Objects in 100s of playlists)
- asdff 4y agoI don't know how people deal with tags. It adds so much friction to me. Naming tags, deciding what rules this tag is supposed to have, deciding what stuff is tagged. I tried the firm approach of being extremely discrete with tags and it took a lot of effort, and I've tried the loose approach of tagging things if they are even slightly related which imo defeated the whole purpose of organizing things to make it easy to find them later if a lot of tangentially related things share the same tags. Folders seem a lot more straightforward for me at least, and if I need something in two places at once, there's always ln -s
- flanked-evergl 4y agoLook at wikidata, RDF and semantic web. This is somewhat a well solved problem that should not be solved differently again.
- deleted 4y ago[deleted]
- feoren 4y agoI'm surprised I haven't seen more discussion of how tags are an entry point into plain-old data architecture. It should be obvious that by the time you're using tags for queries like "start-date: BEFORE 2022-03-01", you've created an inner-platform where you're building a plain-old relational database on top of your tags. Stop what you're doing and elevate "start date" out of tag-land and into a more structured representation with more application support. Many enterprise databases add a memo field called "Comments" to almost every table. Clients very often end up coming up with their own guidelines about how to embed various information in the comments fields that the primary structure is missing. Looking over how clients are using the "comments" fields is a great way to discover new things that should be formally incorporated into the structure of your data architecture. Similarly with tags. Look at tags as a starting point for adding a bit of loose structure to the frontiers of your data architecture. Mix them in with more structured data architecture. Be ready to "graduate" tags up to the next level of structure when it becomes appropriate. Stop worrying about how to make tagging perfect and embrace it for what it is: an easy way to get started on modeling the parts of the domain that you haven't spent a long time thinking about yet. A good way to understand how users want to use your system. Something you're always revisiting, cleaning up, and using as a source of inspiration. If you see some tags getting out of hand, don't try to improve your tagging system; instead take what those tags are trying to represent and add more structured fields and queries for them. This pipeline of less to more structure should be constantly playing out in a healthy, evolving system.
- sebastialonso 4y agoWhat a weird decision to store dates in tags, it looks like bad design. Fully agree with the primer to data architecture, I've seen people almost getting to the point of writing DSL over tags. Madness!
- physicles 4y agoAs someone who dabbled in adding basic tagging to a database recently, this take feels absolutely spot-on. I’ll draw on it when evolving our system forward.
- jon-wood 4y ago
- fleddr 4y agoAs many commenters have mentioned (as does the article) hierarchical tags are a pain, if not an impossibility to get right. Related tags, though, can be done on the cheap and are surprisingly powerful, fun and cool under the right conditions. Say you have a massive database of photos, each photo having tags. As example we'll use the tag "United States", which is used as a tag on 50,000 photos. Next, you go over each of those 50,000 photos and check which other tags were used, and sort them by occurrence. This reveals useful and often surprising implicit relations between tags. The relation can be of any type, hierarchical or otherwise. It reveals relations never explicitly mapped or maintained. It's organic, which kind of fits the philosophy of tagging.
- tinco 4y agoIf you're at the point where you're adding hierarchies to your tags, I think you're fighting a losing battle. At that point, why not do what Google does and just make a BERT embedding. No way you're going to manually achieve the full extent of complexity of how humans group and describe things.
- acchow 4y agoSounds like they are trying to embed the search semantics in the data storage. Why not treat search as a distinct problem?
- redbar0n 4y agoYes. Clicking a tag is like searching for a single word. The crux is that tags add useful metadata that may not be in the content it is supposed to tag. Maybe instead of modeling tags separately from content, metadata should simply be joined into the content itself at the end, and then searched using the same text search tools used for content?
- CobrastanJorji 4y agoOne weird content tagging system I recall was Amazon's "Amapedia" (https://en.wikipedia.org/wiki/Amapedia https://en.wikipedia.org/wiki/Amapedia). It was a product wiki, a way for people to curate information of all sorts about Amazon products. It allowed each product to be arbitrarily tagged. It was short-lived, failed, and abandoned, for all of the reasons you'd immediately expect. What was neat about it was that it must have involved someone a little too interested in set theory. A product was an article, and a product could have tags, but tags were themselves articles, and so tags could also have tags, and those tags could also belong to tags, etc. The whole system was focused on these tags. If you wanted to compare two products, you'd compare the pages, and the comparison would focus on the differences in the tags of the two pages. Tags could have values, too, so products could have a "RAM" tag, and each RAM tag would have an associated value for that page, but the RAM page itself would have general information about RAM as a concept (which would probably have tags itself...). Searching worked the same way. You could search for pages with certain tags or tags whose values were greater/less/equal to whatever values. Anyway, it was a fun and interesting way to do content tagging that did not work out.
- scottmcdot 4y agoYet we still can't search for multiple hash tags on instagram.
- didgetmaster 4y agoI created a new kind of object store where tagging is one of its key features. Each data object (called a Didget - short for Data Widget) can have a set of contextual tags attached. This is true whether the Didget holds file data like a photo, a document, or a piece of software; or if it holds other kinds of structured or semi-structured data (relational tables, folders, configuration, etc.). Each defined tag has a data type (STRING, INTEGER, DATETIME, etc.) and a 2 level context. Like a column in a relational table within a columnar store; all the values for the same defined tag are stored together. This makes querying extremely fast. So you can define tags like Person.FirstName, Event.Wedding, FileSystem.Extension and then attach values to files and other kinds of content. You can then query the system (e.g. Find all photos where Person.FirstName = 'Billy') based on their tags. I have created containers with 200M of these objects and put a dozen or so tags on each one. It can run queries that return in just a couple of seconds. Demo Video: https://www.youtube.com/watch?v=dWIo6sia_hw https://www.youtube.com/watch?v=dWIo6sia_hw
- NWoodsman 4y agoIs your design open-source? Do you have an API? Would like to learn/help
- ethn 4y agoGenerally you either use Latent Dirichlet Allocation, exact tags, or a mixture of both. I structure the metric space to weigh exact tags greater than LDA—-whereas you can then create two more classes in that LDA space, of the heavier similar tags and then the description.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- maoeurk 4y agoI've been wanting to make a datalog tagging system for a few things for a while now but don't have the energy to actually do it. Essentially the idea is to encode relationships allowing for very specific queries like: "show me pictures of a person wearing a green hat looking at another person" which is not something most tagging systems could reasonably do. Breaking that down, that'd be something like: wearing(person1, hat), is_hat(hat), is_green(hat), is_person(person1), is_person(person2), looking_at(person1, person2). I wanted to apply this to Brazilian Jiu Jitsu videos to be able to find very specific queries like, "matches where player 1 gets a takedown, gets swept by player 2, and player 2 wins by submission". A sufficiently well tagged data set would let you find specific stories and sequences of events in a way that I don't think a non-computational query system could do. Most of the effort and value around a system like this would be building a community of people to tag the data and tools to make that tagging easy... and perhaps a more user friendly query UI.
- youainti 4y agoSo you might be interested to know that medical information is described in the way that you propose. Snowmed CT [0] uses a standardized set of "relationships" between "concepts" [0] https://en.wikipedia.org/wiki/SNOMED_CT?wprov=sfla1 https://en.wikipedia.org/wiki/SNOMED_CT?wprov=sfla1
- hamasho 4y agoWhen I started creating a simple blog system as a newbie developer, I needed to design its category/tagging system. Then I was surprised by the lack of good resources on how to design such basic features. I just wanted to know several design patterns and their pros and cons, but I couldn't find any, so I ended up designing my own crappy system. I hope someone wrote articles on it with actual DB schemas.
- c7b 4y agoTags are arguably superior to folders for organising files, unfortunately the major OS don't seem to agree. I'm using the same (expanded, adapted) folder structure for all my files since I got my first computer, and it's survived multiple OS migrations, being synced between multiple devices with different form factors, multiple changes in life circumstances (school, undergrad, postgrad, work),... I love tags and I've used them in some parts (eg in my old mp3 collection, for academic papers, for Anki flash cards) and I'd love to use a (simple and dumb, not rich enough to enable set theory paradoxes) tagging system to organise my files instead. However, my experience has left me convinced that the only truly long-term solution for your personal data are flat files sitting on your hard drive inside a simple hierarchical folder structure. Anything else is likely going to rot at some point, after a system change, after some BigTech decides they want to use something else, after a start-up disappears, or it's going to keep you locked into some walled garden. Unless there's something I've missed, if so please let me know.
- ec2y 4y agoVery true. That’s why I’ve got endnote as my digital card catalog for papers. (Yes, there may be better reference tools but I’m too invested at this point…)
- cranium 4y agoTags are beautiful. They enable a non-hierarchical way of linking elements together so they form a graph. And graphs are beautiful. But they are also messy and bring a whole cohort of problems that you wouldn't have with trees. The problem with tags is that they are the first and often only metadata available to represent the complex relationships between elements. So everything goes in it: tags for the semantic (ontology is rabbit hole in itself), tags for relations with other items, and not forgetting the tags project management (priorities, people, milestone,...). Want to empower your tags, for instance adding hierarchy or dynamic tags? Then every tag will get these features and associated problems. A solution would be to have tags of different "types", each processed differently, and migrate the metadata from a "bag of tags" to "a bag of bags of tags". But then tagging wouldn't be as simple writing a name in a field.
- kristianp 4y agoIn a more readable form: https://threadreaderapp.com/thread/1534301374166474752.html https://threadreaderapp.com/thread/1534301374166474752.html
- nickm12 4y agoA lot of the author's questions can be answered by "use an inverted index".
- 987655321 4y ago