4 ms·

A few months ago I worked on some proof-of-concept code for searching tagged data: https://github.com/aaviator42/Cha https://github.com/aaviator42/Cha I now wo

A few months ago I worked on some proof-of-concept code for searching tagged data: https://github.com/aaviator42/Cha https://github.com/aaviator42/Cha I now work full-time in a role where part of my duties is designing a content tagging system and its search functionalities. It's very interesting and fun! Lots of puzzles. How do you weigh different tags? How do you do fuzzy searching ('city' should match with plural ('cities'), misspellings ('citys'), etc)? How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? What about synonyms? What about regional terminology and synonym tables? Then there's one-to-one and one-to-many and many-to-one mapping. As a side project I'm also working on a beta public search engine that I'll launch on HN sometime in the next year or so, where I'm having similar puzzles.

cube2222 4y ago
> How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? That sounds like a very good use case for word embeddings.
- bell-cot 4y ago
  How do you deal with "hotdog" possibly being a noun (several meanings), or proper noun (several meanings), or verb, or interjection?
  - dmonitor 4y ago
    e621 frequently has to deal with characters with the same name, or an artist with the same name as a character. they just make ambiguous tags have a special syntax. so if bob was an artist, but also had a character named bob, it would just be bob_(bob) for the character and bob_(artist) for the artist. and if someone tried to tag something as just “bob” they would be told to be more specific. searching for all bobs can be done with bob_(*). so hotdog could have hotdog_(food), hotdog_(interjection), and hot dogs (the animal) would be two tags: hot and dog. it’s not the cleanest solution, but it works well enough.
papito 4y ago
Seems to me like a lot of these are solved by a dedicated search engine? I see it in this business all the time how people try to reinvent the wheel and end up writing their own search engine, thinking it's just another small in-house project, but they quickly run into the difficult problems like, well, these.
pkd 4y ago
Nice. Your Cha project has the beginnings of a search engine with cosine similarity, even though I don't think you intend to take it there. Tagged items are like a precomputed inverted index, and the matching is a search on that index.
- aaviator42 4y ago
  Yep! I'm not planning on working further on Cha at the moment, but I've learnt a lot since I initially wrote the PoC code and that is probably how I'd go about it.

cube2222 4y ago

> How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? That sounds like a very good use case for word embeddings.

bell-cot 4y ago
How do you deal with "hotdog" possibly being a noun (several meanings), or proper noun (several meanings), or verb, or interjection?
- dmonitor 4y ago
  e621 frequently has to deal with characters with the same name, or an artist with the same name as a character. they just make ambiguous tags have a special syntax. so if bob was an artist, but also had a character named bob, it would just be bob_(bob) for the character and bob_(artist) for the artist. and if someone tried to tag something as just “bob” they would be told to be more specific. searching for all bobs can be done with bob_(*). so hotdog could have hotdog_(food), hotdog_(interjection), and hot dogs (the animal) would be two tags: hot and dog. it’s not the cleanest solution, but it works well enough.

papito 4y ago

Seems to me like a lot of these are solved by a dedicated search engine? I see it in this business all the time how people try to reinvent the wheel and end up writing their own search engine, thinking it's just another small in-house project, but they quickly run into the difficult problems like, well, these.

pkd 4y ago

Nice. Your Cha project has the beginnings of a search engine with cosine similarity, even though I don't think you intend to take it there. Tagged items are like a precomputed inverted index, and the matching is a search on that index.

aaviator42 4y ago
Yep! I'm not planning on working further on Cha at the moment, but I've learnt a lot since I initially wrote the PoC code and that is probably how I'd go about it.