6 ms·

Tag Systems

109 points by fagnerbrack 4y ago

Rygian 4y ago
One very prominent use of Key-value tags is the OpenStreetMap database. The datamodel is deceptively simple: nodes, ways, and relations to represent geographical locations, and a list of key-values attached to any of them to define the semantics.
2sk21 4y ago
One thought that has occurred to me when using tagging systems is that they ought to have the ability to refactor tags. No matter what the tagging scheme is initially, it will surely mutate over time. It should be possible to rename tags, join tags and divide one tag into multiple tags. And while you are at it, perhaps it could use some kind of sentence embeddings to suggest tags that mean approximately the same thing.
- flir 4y ago
  Stemming would get you 95% of the way there in coalescing tags. Would sentence embedding outperform that significantly?
- nerdponx 4y ago
  In my opinion this is largely a UI/UX issue. For some reason, applications with rich tagging systems have only the most raw basic interface for manipulating and searching using tags.
  - benrbray 4y ago
    I think something like datalog queries would be perfect for refactoring and searching tags. Would also let you declare relationships between tags, like "tagged(x, calculus) -> tagged(x, math)"
deafpolygon 4y ago
Look, you can be a horse girl if you love horses. You don't need to own one to be a horse girl. Why gatekeep, huh?
- esafak 4y ago
  I believe you are referring to the "is" vs "has" distinction.
  - deafpolygon 4y ago
    Polymorphism or composition?
oneeyedpigeon 4y ago
> “should users be able to query a specific tag alias?” ie, can I search for stories tagged with the alias snarry? There’s no correct choice here. AO3 went with “no”. This is a very interesting point. My initial thought was: there definitely IS a correct choice, which is "no". If the choice were "yes", that would imply there's a difference between the tags, but they're aliases, so there shouldn't be a difference. An alias should be exactly interchangeable. Then I thought about how practically impossible that is to enforce, given the nature of language. The usage choice itself implies a difference; there are no true aliases in language.
- zamnos 4y ago
  I think it depends on the usage is tags, and how important the tagging system actually is, because there's a tagging system in wide use that allows for aliases. That system is reactj on Slack messages. For those outside the slack ecosystem, that feature allows you to an emoji to a message, even or especially if you're not the author. Thing is, as tiny pictures, people remember what they remember about the emoji, and people often remember an alias and may not even know the offical name. Slack further allows customization by letting users add their own reactj imbuing each slack workspace with its own personality. If someone wants to type in dumbbell and someone wants to type in weight, and it goes to the same tiny picture, is that really so broken?
  - oneeyedpigeon 4y ago
    It always bugs me that the default alias for the fist bump emoji is "facepunch"... I think we're ultimately in agreement — aliases should be 'transparent' and not imbued with any specific meaning. Of course, it's much easier for a very limited system like emoji to be able to deal with these things than an open-ended tagging system, especially one that caters for the entirety of Unicode.
- at_a_remove 4y ago
  AO3 has many, many decisions around tagging that are quirky, at best.
somat 4y ago
If you think hard enough about tag systems you end up with the idea that the unix filesystem is a nearly ideal way to tag data. My hierarchy of tag systems goes something like key, key=value, hierarchical key(the unix filesystem), hierarchical key=value(oh shit you found ldap, abort, you went to far)
- giantrobot 4y ago
  Iff you assume the use of symbolic links. A tagging system needs the ability to have arbitrary tags and the same item referenced by multiple tags.
  - zamnos 4y ago
    If you're on one file system, hard links work as well. They have they added benefit that you can delete an entire tree but files that are linked to from elsewhere won't get deleted or end up with dangling refs.
  - somat 4y ago
    A description of the unix file system but using tag vernacular instead of the more common forms. a file is a generic unformed array of bytes, the application can store anything it wants in this array. files are selected via tags, a file can have any number of tags, however there is a limitation that each tag can only reference one file. tag prefixes are use to group and match common tags.
Gehinnn 4y ago
A tag system is actually a deterministic computational model https://en.m.wikipedia.org/wiki/Tag_system https://en.m.wikipedia.org/wiki/Tag_system
- hwayne 4y ago
  Implementable in vim keystrokes! https://buttondown.email/hillelwayne/archive/vim-is-turing-complete/ https://buttondown.email/hillelwayne/archive/vim-is-turing-c...
DontchaKnowit 4y ago
Can anyone tell me if anything like this exists : A program that runs in the background on your machine and watches processes and the file handles they open. Then, keeps a database of associations between files and programs. E.g. you open a pdf with adobe acrobat, and now it adds an entry with a relation between that specific file and acrobat, and between pdf file extensions and acrobat. Then, this information is used to provide an intuitive interface into the file system. Searching for that .docx file you made last year and cant remember for the life of you where you put it? Just search for .docx, or files associated with "Word" This to me is the holy grail of filesystem search, and it doesn't seem that hard to implement, but I've never found anything like it.
- epilys 4y ago
  I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux: Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx). Associate PIDs that open those files with their .desktop entries[1] Store results in some database like sqlite3.[2] Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3]. I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper https://github.com/javierhonduco/sweeper [0]: https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace [1]: https://www.freedesktop.org/wiki/Specifications/desktop-entry-spec/ https://www.freedesktop.org/wiki/Specifications/desktop-entr... [2]: https://sqlite.org/ https://sqlite.org/ [3]: https://developer.gnome.org/documentation/tutorials/search-provider.html https://developer.gnome.org/documentation/tutorials/search-p...
  - DontchaKnowit 4y ago
    This is wonderful, thank you. I may try this out. Originally wanted to do it for windows but I eventually concluded it would take some prodigious hackery to get it do work
didgetmaster 4y ago
Good information that deals with tagging issues that I have been thinking about and implementing for several years. I have been working on a new kind of data management system that is designed to replace traditional file systems while also doing a bunch of database operations. It is an object store that makes extensive use of tagging. Nearly all file metadata comes in the form of tags attached to each file. Even its name is a tag, so every file can have multiple names (each object's unique ID is a 64 bit number). The system is currently in beta and still needs work, but it can do some amazing things so far. It can create 100M+ files and attach dozen of tags to each one (there is a limit of 255 tags per object). Searching for all photos with certain tags, for example, can give you the results extremely fast. Even with containers with 100 million objects can give you the 100,000 that have a specific tag in just a second or two. Tags have context. Instead of 'John' as a tag, you attach Person.FirstName = 'John' to photos of John or documents written by John. The whole system is like a sparse relational database table where each row/column intersection can have multiple values. We are looking for more beta testers and anyone can download and try the software for free. Here are a couple short videos showing a few things it can do. https://www.youtube.com/watch?v=yLPNLHm9fIk https://www.youtube.com/watch?v=yLPNLHm9fIk https://www.youtube.com/watch?v=dWIo6sia_hw https://www.youtube.com/watch?v=dWIo6sia_hw
- zvr 3y ago
  This is wonderful, but unfortunately only for Windows. If you want beta testers for a Linux version... I'm more interested in the Manager with an API, not the GUI Browser.
- runlaszlorun 4y ago
  Nice. I’ve wanted something like this for 15 years. I’m curious how you implemented them. I’ve been mulling using a tag based system for a completely unrelated project. I figure one might get a lot of way there using hash tables and sets from any of the high level languages. But it seems you maybe used a different approach?
  - didgetmaster 4y ago
    Each logical container (a Pod) has a set of data objects called Didgets (short for Data Widgets) where each one is assigned a unique 64-bit number as its ID. Each tag is defined with a data type (e.g. Person.FirstName has type STRING) using a schema like how a column in a relational table is defined. All tags within the same context are stored together in a highly optimized Key-Value store (also stored within specialized Didgets) like how a columnar store database stores all the data for a single column separately. So if you attach a tag like Person.FirstName = 'John' to a photo of John that is stored in a Didgets with ID = 123; then within the tag the value of 'John' is mapped to the key 123 within the Person.FirstName tag store. This makes it extremely efficient to find all the Didgets where 'John' was attached as a person's first name. It can also find everything where the first name starts with 'J' or a vowel (using regex). The values within each tag store are de-duplicated and reference counted. In the example, the value 'John' is stored once. If 100 things have 'John' attached, then its reference count is 100. This makes it extremely fast to find everything with the 10 most popular names. The tags were so efficient and fast that I built relational tables using them that are often faster to query than traditional database systems. In this case the 'Key' each value is mapped to is the rowID within the table. Here is a video showing how that compares to SQLite: https://www.youtube.com/watch?v=Va5ZqfwQXWI https://www.youtube.com/watch?v=Va5ZqfwQXWI
dsr_ 4y ago
I am not enamored of tags, because tag systems break down when humans get involved. The same person will choose different sets of tags for the same file, when asked on different days; they will end up with synonymous tags that should be coalesced, but nobody wants to do that drudgework. The semantics of each tag will alter over time and by person. An optimal tag system is inferior to a full text search system with an indexer that recommends keywords and has a reasonable syntax for includes, excludes, and boolean logic.
- valenterry 4y ago
  That does not really speak against tags in general. You can also predefine tags for instance.
- Joker_vD 4y ago
  While I absolutely agree with the downsides you mention in the 2nd and 3rd paragraphs, the first paragraph is kinda trivial: all systems break down when humans get involved. As for the last one... what about images? People tag them too, and no "full text search system" exist for them unless you count "textual inversion"/"interrogate CLIP" which would still requires quite some hefty amount of manual tagging for the initial training.
- cptcobalt 4y ago
  This is exactly what the tag alias system of AO3—as discussed on this page—is designed to mitigate. If you've spent any amount of time on that website, you'll realize their tag system works really, really well.
  - dsr_ 4y ago
    I've seen it. Their tag system relies on a large group of volunteers to reconcile the problems, and all the volunteers abiding by a large set of rules: https://archiveofourown.org/wrangling_guidelines/11 https://archiveofourown.org/wrangling_guidelines/11
- twic 4y ago
  The way i would frame this is that tag systems have a steep return on effort expended. Low-effort tagging is garbage, as you have observed. But when users stay on top of it for some reason, it can be extremely valuable. Someone else mentioned the example of AO3 here.
egeozcan 4y ago
The best way is, in my experience, let the users tag away. Then add the tags to the item descriptions in the full text search (while also holding on to the normalized data, because refactoring tags is a thing). If your FTS system allows the definition of synonyms, even better. Users get the usual autocomplete, and suggested tags. Searchers get approximation. I never implemented this before but the tags can also hold references to the tags they are most frequently tagged with to further add content to the depleted search results. Never thought how the implementation would look like though.
joshu 4y ago
at the very beginning del.icio.us had a hierarchy operator: anything tagged "foo/bar" would internally also get the tag "foo" as well. i ended up pulling it out. for finding untagged things, i had "system:untagged" automatically added if the items had no tags because mixing inner and outer joins was a nightmare in mysql.
codr7 4y ago
I've slowly drifted into preferring tags over log levels. Strict hierarchies always turn into dead ends sooner or later. https://github.com/codr7/cl-redb https://github.com/codr7/cl-redb
- codr7 4y ago
  Edit: https://github.com/codr7/cl-slog https://github.com/codr7/cl-slog
Night_Thastus 4y ago
I love a good tagging system, if it is done properly. It makes it so effortless to find the content you want. I broadly like this post, but I disagree with "The lesson: don’t let users encode logical paradoxes.". If a user says they want (A && !A), just hand them back nothing. Let the expressiveness exist regardless. (Ideally, warn the user, though) I want expressive tags. It makes everything so much easier. If I can't do (A && B) || !(C or D) then as a user I'm going to get frustrated if there's a lot of content to sift through. Grouping and negation are critical, IMO. The post touches on this, but I'd say by far the largest issue is the moderation of tags. If only the author is allowed to tag, and there's no moderation or oversight from other users or official moderators, the tag system is effectively useless. Some users will be lazy and not tag effectively. Others will tag incorrectly by mistake or intention. You want the system to support new tags for a variety of reasons, but this allows typos to create new tags. Generally, when I see that a tag system sucks, it's because it's been mis-used, not because the underlying logic is bad.
- bccdee 4y ago
  The paradox they were referring to was A := (B), B := !(A). That's not a expressiveness issue, that's just a dependency cycle—there's no reason to allow it.
  - Night_Thastus 4y ago
    Oh, assignment? Yeah, that's likely going too far. I completely misunderstood the issue they were stating.
codemac 4y ago
The worst part about going from G to FB was going from buganizer to fbtasks. The internal tool at G has a strict hierarchy called "components", and fbtasks is tags alone. This means if you want to file a bug against anything at G, you could navigate this large hierarchy and find a rough spot for the bug. Regardless of that team's documentation, tooling, bug tracking helper forms, etc you could always file a bug that would get pretty close. Then someone on that team would move it into the proper component if you got it wrong. At FB... you go to file a bug, but you need to know which tags! If you miss the right tag, that team may never see it. All teams implement their tagging differently, and many implement almost nothing at all. This devolves to pretty much all bugs being tracked through internal group posts, because at least internal groups have names and membership lists that are easily searchable. Then teams implement bots & programs to take every post filed in a group, and auto generate a task with the right tags! And pray that they append the `donotreap` tag... Bugs need to be filed against specific systems, and those systems need to be owned by teams. Those teams are in orgs, those orgs are in companies. I cringe whenever I see a bug tracker for a large company that is not in a hierarchy.
- dietr1ch 4y ago
  You could have a tag hierarchy, or even multiple ones. I don't think tags prevent you from navigating the hierarchy, it's just that at some point you need to build the hierarchies and make them accessible, and just having tags won't push you there.
kayo_20211030 4y ago
From 2001, and still the same. https://people.well.com/user/doctorow/metacrap.htm https://people.well.com/user/doctorow/metacrap.htm
- didgetmaster 4y ago
  >2.5 Schemas aren't neutral Nowhere is this more prominent than in traditional file systems where the folder hierarchy is often used as a kind of metadata tagging system. You take a picture while on vacation and copy it to your /pictures/2020/vacations/Hawaii/beach folder. But you could also copy it to your /2020/vacations/Hawaii/beach/pictures folder. Your pictures, documents, music, and source code gets spread out using whatever hierarchy seems right to you on the day you copied or created it. Heaven forbid you decide to change the folder structure using anything but symbolic links so it breaks any stored references (i.e. full path) to the file(s).
remram 4y ago
I think an important part of tagging systems is that you want some way to store what tags have been considered. Like, this story has "female-protagonist" but not "teen-protagonist": someone clearly looked at the characters and applied tags, but does that mean it does not have a teen protagonist? Maybe the tag was created after someone reviewed the story in order to apply "*-protagonist" tags. Maybe that person was only doing "female-protagonist" in bulk. Having a system that can point out unknowns is useful both for search (by including them in results) and for filling in that data (by suggesting those stories for review).
junon 4y ago
Then there's the problem of synchronization. How do you take two bodies of tagged data and merge them? How do you create a degree of certainty that the tags themselves have any similarity?

109 points by fagnerbrack 4y ago

One very prominent use of Key-value tags is the OpenStreetMap database. The datamodel is deceptively simple: nodes, ways, and relations to represent geographical locations, and a list of key-values attached to any of them to define the semantics.

2sk21 4y ago

One thought that has occurred to me when using tagging systems is that they ought to have the ability to refactor tags. No matter what the tagging scheme is initially, it will surely mutate over time. It should be possible to rename tags, join tags and divide one tag into multiple tags. And while you are at it, perhaps it could use some kind of sentence embeddings to suggest tags that mean approximately the same thing.

flir 4y ago
Stemming would get you 95% of the way there in coalescing tags. Would sentence embedding outperform that significantly?
nerdponx 4y ago
In my opinion this is largely a UI/UX issue. For some reason, applications with rich tagging systems have only the most raw basic interface for manipulating and searching using tags.
- benrbray 4y ago
  I think something like datalog queries would be perfect for refactoring and searching tags. Would also let you declare relationships between tags, like "tagged(x, calculus) -> tagged(x, math)"

deafpolygon 4y ago

Look, you can be a horse girl if you love horses. You don't need to own one to be a horse girl. Why gatekeep, huh?

esafak 4y ago
I believe you are referring to the "is" vs "has" distinction.
- deafpolygon 4y ago
  Polymorphism or composition?

oneeyedpigeon 4y ago

> “should users be able to query a specific tag alias?” ie, can I search for stories tagged with the alias snarry? There’s no correct choice here. AO3 went with “no”. This is a very interesting point. My initial thought was: there definitely IS a correct choice, which is "no". If the choice were "yes", that would imply there's a difference between the tags, but they're aliases, so there shouldn't be a difference. An alias should be exactly interchangeable. Then I thought about how practically impossible that is to enforce, given the nature of language. The usage choice itself implies a difference; there are no true aliases in language.

zamnos 4y ago
I think it depends on the usage is tags, and how important the tagging system actually is, because there's a tagging system in wide use that allows for aliases. That system is reactj on Slack messages. For those outside the slack ecosystem, that feature allows you to an emoji to a message, even or especially if you're not the author. Thing is, as tiny pictures, people remember what they remember about the emoji, and people often remember an alias and may not even know the offical name. Slack further allows customization by letting users add their own reactj imbuing each slack workspace with its own personality. If someone wants to type in dumbbell and someone wants to type in weight, and it goes to the same tiny picture, is that really so broken?
- oneeyedpigeon 4y ago
  It always bugs me that the default alias for the fist bump emoji is "facepunch"... I think we're ultimately in agreement — aliases should be 'transparent' and not imbued with any specific meaning. Of course, it's much easier for a very limited system like emoji to be able to deal with these things than an open-ended tagging system, especially one that caters for the entirety of Unicode.
at_a_remove 4y ago
AO3 has many, many decisions around tagging that are quirky, at best.

somat 4y ago

If you think hard enough about tag systems you end up with the idea that the unix filesystem is a nearly ideal way to tag data. My hierarchy of tag systems goes something like key, key=value, hierarchical key(the unix filesystem), hierarchical key=value(oh shit you found ldap, abort, you went to far)

giantrobot 4y ago
Iff you assume the use of symbolic links. A tagging system needs the ability to have arbitrary tags and the same item referenced by multiple tags.
- zamnos 4y ago
  If you're on one file system, hard links work as well. They have they added benefit that you can delete an entire tree but files that are linked to from elsewhere won't get deleted or end up with dangling refs.
- somat 4y ago
  A description of the unix file system but using tag vernacular instead of the more common forms. a file is a generic unformed array of bytes, the application can store anything it wants in this array. files are selected via tags, a file can have any number of tags, however there is a limitation that each tag can only reference one file. tag prefixes are use to group and match common tags.

Gehinnn 4y ago

A tag system is actually a deterministic computational model https://en.m.wikipedia.org/wiki/Tag_system https://en.m.wikipedia.org/wiki/Tag_system

hwayne 4y ago
Implementable in vim keystrokes! https://buttondown.email/hillelwayne/archive/vim-is-turing-complete/ https://buttondown.email/hillelwayne/archive/vim-is-turing-c...

DontchaKnowit 4y ago

Can anyone tell me if anything like this exists : A program that runs in the background on your machine and watches processes and the file handles they open. Then, keeps a database of associations between files and programs. E.g. you open a pdf with adobe acrobat, and now it adds an entry with a relation between that specific file and acrobat, and between pdf file extensions and acrobat. Then, this information is used to provide an intuitive interface into the file system. Searching for that .docx file you made last year and cant remember for the life of you where you put it? Just search for .docx, or files associated with "Word" This to me is the holy grail of filesystem search, and it doesn't seem that hard to implement, but I've never found anything like it.

epilys 4y ago
I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux: Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx). Associate PIDs that open those files with their .desktop entries[1] Store results in some database like sqlite3.[2] Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3]. I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper https://github.com/javierhonduco/sweeper [0]: https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace [1]: https://www.freedesktop.org/wiki/Specifications/desktop-entry-spec/ https://www.freedesktop.org/wiki/Specifications/desktop-entr... [2]: https://sqlite.org/ https://sqlite.org/ [3]: https://developer.gnome.org/documentation/tutorials/search-provider.html https://developer.gnome.org/documentation/tutorials/search-p...
- DontchaKnowit 4y ago
  This is wonderful, thank you. I may try this out. Originally wanted to do it for windows but I eventually concluded it would take some prodigious hackery to get it do work

didgetmaster 4y ago

Good information that deals with tagging issues that I have been thinking about and implementing for several years. I have been working on a new kind of data management system that is designed to replace traditional file systems while also doing a bunch of database operations. It is an object store that makes extensive use of tagging. Nearly all file metadata comes in the form of tags attached to each file. Even its name is a tag, so every file can have multiple names (each object's unique ID is a 64 bit number). The system is currently in beta and still needs work, but it can do some amazing things so far. It can create 100M+ files and attach dozen of tags to each one (there is a limit of 255 tags per object). Searching for all photos with certain tags, for example, can give you the results extremely fast. Even with containers with 100 million objects can give you the 100,000 that have a specific tag in just a second or two. Tags have context. Instead of 'John' as a tag, you attach Person.FirstName = 'John' to photos of John or documents written by John. The whole system is like a sparse relational database table where each row/column intersection can have multiple values. We are looking for more beta testers and anyone can download and try the software for free. Here are a couple short videos showing a few things it can do. https://www.youtube.com/watch?v=yLPNLHm9fIk https://www.youtube.com/watch?v=yLPNLHm9fIk https://www.youtube.com/watch?v=dWIo6sia_hw https://www.youtube.com/watch?v=dWIo6sia_hw

zvr 3y ago
This is wonderful, but unfortunately only for Windows. If you want beta testers for a Linux version... I'm more interested in the Manager with an API, not the GUI Browser.
runlaszlorun 4y ago
Nice. I’ve wanted something like this for 15 years. I’m curious how you implemented them. I’ve been mulling using a tag based system for a completely unrelated project. I figure one might get a lot of way there using hash tables and sets from any of the high level languages. But it seems you maybe used a different approach?
- didgetmaster 4y ago
  Each logical container (a Pod) has a set of data objects called Didgets (short for Data Widgets) where each one is assigned a unique 64-bit number as its ID. Each tag is defined with a data type (e.g. Person.FirstName has type STRING) using a schema like how a column in a relational table is defined. All tags within the same context are stored together in a highly optimized Key-Value store (also stored within specialized Didgets) like how a columnar store database stores all the data for a single column separately. So if you attach a tag like Person.FirstName = 'John' to a photo of John that is stored in a Didgets with ID = 123; then within the tag the value of 'John' is mapped to the key 123 within the Person.FirstName tag store. This makes it extremely efficient to find all the Didgets where 'John' was attached as a person's first name. It can also find everything where the first name starts with 'J' or a vowel (using regex). The values within each tag store are de-duplicated and reference counted. In the example, the value 'John' is stored once. If 100 things have 'John' attached, then its reference count is 100. This makes it extremely fast to find everything with the 10 most popular names. The tags were so efficient and fast that I built relational tables using them that are often faster to query than traditional database systems. In this case the 'Key' each value is mapped to is the rowID within the table. Here is a video showing how that compares to SQLite: https://www.youtube.com/watch?v=Va5ZqfwQXWI https://www.youtube.com/watch?v=Va5ZqfwQXWI

dsr_ 4y ago

I am not enamored of tags, because tag systems break down when humans get involved. The same person will choose different sets of tags for the same file, when asked on different days; they will end up with synonymous tags that should be coalesced, but nobody wants to do that drudgework. The semantics of each tag will alter over time and by person. An optimal tag system is inferior to a full text search system with an indexer that recommends keywords and has a reasonable syntax for includes, excludes, and boolean logic.

valenterry 4y ago
That does not really speak against tags in general. You can also predefine tags for instance.
Joker_vD 4y ago
While I absolutely agree with the downsides you mention in the 2nd and 3rd paragraphs, the first paragraph is kinda trivial: all systems break down when humans get involved. As for the last one... what about images? People tag them too, and no "full text search system" exist for them unless you count "textual inversion"/"interrogate CLIP" which would still requires quite some hefty amount of manual tagging for the initial training.
cptcobalt 4y ago
This is exactly what the tag alias system of AO3—as discussed on this page—is designed to mitigate. If you've spent any amount of time on that website, you'll realize their tag system works really, really well.
- dsr_ 4y ago
  I've seen it. Their tag system relies on a large group of volunteers to reconcile the problems, and all the volunteers abiding by a large set of rules: https://archiveofourown.org/wrangling_guidelines/11 https://archiveofourown.org/wrangling_guidelines/11
twic 4y ago
The way i would frame this is that tag systems have a steep return on effort expended. Low-effort tagging is garbage, as you have observed. But when users stay on top of it for some reason, it can be extremely valuable. Someone else mentioned the example of AO3 here.

egeozcan 4y ago

The best way is, in my experience, let the users tag away. Then add the tags to the item descriptions in the full text search (while also holding on to the normalized data, because refactoring tags is a thing). If your FTS system allows the definition of synonyms, even better. Users get the usual autocomplete, and suggested tags. Searchers get approximation. I never implemented this before but the tags can also hold references to the tags they are most frequently tagged with to further add content to the depleted search results. Never thought how the implementation would look like though.

joshu 4y ago

at the very beginning del.icio.us had a hierarchy operator: anything tagged "foo/bar" would internally also get the tag "foo" as well. i ended up pulling it out. for finding untagged things, i had "system:untagged" automatically added if the items had no tags because mixing inner and outer joins was a nightmare in mysql.

codr7 4y ago

I've slowly drifted into preferring tags over log levels. Strict hierarchies always turn into dead ends sooner or later. https://github.com/codr7/cl-redb https://github.com/codr7/cl-redb

codr7 4y ago
Edit: https://github.com/codr7/cl-slog https://github.com/codr7/cl-slog

Night_Thastus 4y ago

I love a good tagging system, if it is done properly. It makes it so effortless to find the content you want. I broadly like this post, but I disagree with "The lesson: don’t let users encode logical paradoxes.". If a user says they want (A && !A), just hand them back nothing. Let the expressiveness exist regardless. (Ideally, warn the user, though) I want expressive tags. It makes everything so much easier. If I can't do (A && B) || !(C or D) then as a user I'm going to get frustrated if there's a lot of content to sift through. Grouping and negation are critical, IMO. The post touches on this, but I'd say by far the largest issue is the moderation of tags. If only the author is allowed to tag, and there's no moderation or oversight from other users or official moderators, the tag system is effectively useless. Some users will be lazy and not tag effectively. Others will tag incorrectly by mistake or intention. You want the system to support new tags for a variety of reasons, but this allows typos to create new tags. Generally, when I see that a tag system sucks, it's because it's been mis-used, not because the underlying logic is bad.

bccdee 4y ago
The paradox they were referring to was A := (B), B := !(A). That's not a expressiveness issue, that's just a dependency cycle—there's no reason to allow it.
- Night_Thastus 4y ago
  Oh, assignment? Yeah, that's likely going too far. I completely misunderstood the issue they were stating.

codemac 4y ago

The worst part about going from G to FB was going from buganizer to fbtasks. The internal tool at G has a strict hierarchy called "components", and fbtasks is tags alone. This means if you want to file a bug against anything at G, you could navigate this large hierarchy and find a rough spot for the bug. Regardless of that team's documentation, tooling, bug tracking helper forms, etc you could always file a bug that would get pretty close. Then someone on that team would move it into the proper component if you got it wrong. At FB... you go to file a bug, but you need to know which tags! If you miss the right tag, that team may never see it. All teams implement their tagging differently, and many implement almost nothing at all. This devolves to pretty much all bugs being tracked through internal group posts, because at least internal groups have names and membership lists that are easily searchable. Then teams implement bots & programs to take every post filed in a group, and auto generate a task with the right tags! And pray that they append the `donotreap` tag... Bugs need to be filed against specific systems, and those systems need to be owned by teams. Those teams are in orgs, those orgs are in companies. I cringe whenever I see a bug tracker for a large company that is not in a hierarchy.

dietr1ch 4y ago
You could have a tag hierarchy, or even multiple ones. I don't think tags prevent you from navigating the hierarchy, it's just that at some point you need to build the hierarchies and make them accessible, and just having tags won't push you there.

kayo_20211030 4y ago

From 2001, and still the same. https://people.well.com/user/doctorow/metacrap.htm https://people.well.com/user/doctorow/metacrap.htm

didgetmaster 4y ago
>2.5 Schemas aren't neutral Nowhere is this more prominent than in traditional file systems where the folder hierarchy is often used as a kind of metadata tagging system. You take a picture while on vacation and copy it to your /pictures/2020/vacations/Hawaii/beach folder. But you could also copy it to your /2020/vacations/Hawaii/beach/pictures folder. Your pictures, documents, music, and source code gets spread out using whatever hierarchy seems right to you on the day you copied or created it. Heaven forbid you decide to change the folder structure using anything but symbolic links so it breaks any stored references (i.e. full path) to the file(s).

remram 4y ago

I think an important part of tagging systems is that you want some way to store what tags have been considered. Like, this story has "female-protagonist" but not "teen-protagonist": someone clearly looked at the characters and applied tags, but does that mean it does not have a teen protagonist? Maybe the tag was created after someone reviewed the story in order to apply "*-protagonist" tags. Maybe that person was only doing "female-protagonist" in bulk. Having a system that can point out unknowns is useful both for search (by including them in results) and for filling in that data (by suggesting those stories for review).

junon 4y ago

Then there's the problem of synchronization. How do you take two bodies of tagged data and merge them? How do you create a degree of certainty that the tags themselves have any similarity?