3 ms·
Search is a pretty solved problem if you are willing to invest the resources to create a inverted index of all the text you want to search. An inverted index of
by UltraSane 1y ago
Search is a pretty solved problem if you are willing to invest the resources to create a inverted index of all the text you want to search. An inverted index of all tweets would be pretty expensive. Creating text embeddings for semantic search would be the next stage and even more expensive.
- lokar 1y agoBasic term based retrieval has been solved for 30+ years The problem is ranking and relevance
- lokar 1y agoThinking more, I imagine each post has limited value for ranking. You need the context of the thread, re-posts, even other threads nearby in time (with the same people).
- simonw 1y agoThey've had an inverted index of all tweets since 2008 (when they acquired Summize). They added a vector index a year and a half ago for a "see related tweets" feature - https://x.com/elonmusk/status/1720314092269822242 https://x.com/elonmusk/status/1720314092269822242 - though as far as I can tell that feature doesn't exist any more, presumably replaced by the ask Grok button.
- phillipcarter 1y agoIt is very much not a solved problem. Because the implication behind search is not "well the result you need is technically in the result set", it's "the result you need as at the top", and that remains an extremely difficult problem for anything but a trivial scale.
- UltraSane 1y agoGood support for regex and boolean operators helps a lot with that. But that requires user skill.
- HeyLaughingBoy 1y agoThen it's not a solved problem in any meaningful way.