4 ms·
So using LLMs to fact-check LLMs with Google results? There are non-LLM fact-checkers out there [1] which aggregate search results from more diverse sources and
by vinni2 3y ago
So using LLMs to fact-check LLMs with Google results? There are non-LLM fact-checkers out there [1] which aggregate search results from more diverse sources and which are shown to outperform LLMs [2].
[1] https://editor.factiverse.ai https://editor.factiverse.ai
[2] https://arxiv.org/abs/2402.12147 https://arxiv.org/abs/2402.12147
- casualscience 3y agoThat literally requires human annotators. If you can do even 75% as well as a human but at 1000x the scale, you still win using the LLMs. Not to mention, I personally do think on a simple task like "Does document A support response B", GPT can probably beat most humans unless the humans are going very slow and possibly have domain knowledge depending on the case.
- genrilz 3y agoI skimmed the paper you linked. Which to be clear to other readers, is a write-up of the technology used in the first link. It looked to me like they use small LLMs to identify factual claims, and rewrite them into search terms. Then the terms are searched using an internal database or search engine, and then LLMs are used to roughly classify how true the sources think the claim is. It looks like a neat product, and trying out the link shows that the sources themselves are presented alongside the classification for the human to peruse, but it looks like it is using LLMs rather heavily. I certainly wouldn't classify it as a non-LLM fact checker.
- Iv 3y agofactiverse.ai states GPT-4 is open source, that there is a colony on Mars and is mixed on whether Finland is a real country. Still needs work