3 ms·

For me the search problem isn't so much about making sure I get back all potentially relevant hits ( more than I could ever read ) , it's how I get the specifi

For me the search problem isn't so much about making sure I get back all potentially relevant hits ( more than I could ever read ) , it's how I get the specific ones I want... So effective search more about excluding than including. Exact phrases or particular keywords are great tools here. Note there is also a difference between finding an answer to a particular question and finding web pages around a particular topic. Perhaps LLM's are more useful for the former - where there is a need to both map the question to an embedding, and summarize the answer - but for the latter I'm not interested in a summary/quick answer, I'm interested in the source material. Sometimes you can combine the two - LLM's for a quick route into the common jargon, which can then be used as keywords.

kkfx 2y ago
> there is also a difference between finding an answer to a particular question and finding web pages around a particular topic. That's exactly the Gessner's problem and unimplementable so far solution: we have information packed in various form, when we want specific bits it's hard do narrow enough available information packages (books, articles, post etc). For common stuff in the vast see of the web we tend to end up very frequently on exactly or nearly exactly what we want, but for less common stuff there is a big problem. Just knowing the historical temperature of a small place where many have recorded temperatures, but still no one have create a table and a graph or daily minima and maxima for let's say the 10 years period we want it's hard to find an answer. For some kind of data we have wikidata, for some others we have public datasets from various sources, but still no easy way to locate and narrow them. LLM are in theory an answer, unfortunately not in practice first for freshness problem (you have to train a model, ingesting new information is still training, far longer than a search engine crawler) and training itself witch is like the opposite of semantic web: instead of having the author prepare the information for machines we have third party who do it en masse, unfortunately they tend to be even less precise than semantic web mean authors PLUS models hallucinations and essentially no ability to tell where their answer came from. That's why we still miss semantic search...

kkfx 2y ago

> there is also a difference between finding an answer to a particular question and finding web pages around a particular topic. That's exactly the Gessner's problem and unimplementable so far solution: we have information packed in various form, when we want specific bits it's hard do narrow enough available information packages (books, articles, post etc). For common stuff in the vast see of the web we tend to end up very frequently on exactly or nearly exactly what we want, but for less common stuff there is a big problem. Just knowing the historical temperature of a small place where many have recorded temperatures, but still no one have create a table and a graph or daily minima and maxima for let's say the 10 years period we want it's hard to find an answer. For some kind of data we have wikidata, for some others we have public datasets from various sources, but still no easy way to locate and narrow them. LLM are in theory an answer, unfortunately not in practice first for freshness problem (you have to train a model, ingesting new information is still training, far longer than a search engine crawler) and training itself witch is like the opposite of semantic web: instead of having the author prepare the information for machines we have third party who do it en masse, unfortunately they tend to be even less precise than semantic web mean authors PLUS models hallucinations and essentially no ability to tell where their answer came from. That's why we still miss semantic search...