3 ms·
We can consider the following when thinking about how to answer this - how do current search engines rank results? - what features correlate with good? - how
by verdverm 4y ago
We can consider the following when thinking about how to answer this
- how do current search engines rank results?
- what features correlate with good?
- how do features of sites get encoded, queried, and returned?
- how might LMs be already involved in this process?
- how might LLMs / ChatGPT like systems encode these things?
- how might we enrich training data to facilitate improved LLM information retrieval?
I would suspect, with current models and algos, that the more often certain statements are found across the training data, the more likely that information is to be "correct"
A good site might be one that has many of these "correct" chunks. Though I doubt they are there yet. A next phase task will likely be along the lines we are discussing here. i.e. How do we return citations and references in responses
- JohnFen 4y ago> how do current search engines rank results? I think there's a poor correlation between search engine rankings and website quality, so it might be better not to include this. > what features correlate with good? Right, that's the essential question. Google's original bit of genius was to to ignore that question entirely, and go with "popular" instead. You can measure popular, but "good" entirely depends on what it is you value. That makes it somewhat subjective.
- verdverm 4y agoPageRank worked well at first by defining quality by the number of backlinks, or sites that linked to it. Essential if you had good content, it was likely that many others pointed to it. Ofcourse this was quickly gamed, but they were able to leverage their search engine usage to observe how users interacted with results, and later pages. I imagine that you could have a quality measurement by finding the chunk embeddings that are most popular, and then finding the sites that have these in the highest concentration. User interaction will still likely be a very important feature in the quality space.
- JohnFen 4y ago> PageRank worked well at first by defining quality by the number of backlinks, or sites that linked to it. Yes. It was measuring popularity. It was genius because it solved the problem usefully well, but page rankings have never been a fantastic indication of quality -- again, depending on what you consider "quality". It was just better than nothing.
- verdverm 4y ago> Yes. It was measuring popularity. Back link counting is not the same thing as popularity, which measures visits PageRank was better than keyword search, which became spammed and PR was resilient to. Now there are 100s of signals being used, see the Yandex leak
- JohnFen 4y agoIt is the same as popularity -- it's measuring the popularity among other websites rather than individuals, but it's popularity nonetheless.
- verdverm 4y agoThey describe it differently in their papar, as importance or quality, as PageRank came out of citation analysis. https://snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom... There's a lot more to the algorithm as well, it's a good read