13 ms·

You don’t need the whole truth and nothing but the truth from it when you are the domain expert. You only need it to highlight areas it thinks are wrong. Also,

by clnq 3y ago

You don’t need the whole truth and nothing but the truth from it when you are the domain expert. You only need it to highlight areas it thinks are wrong. Also, GPT-4 is not 100% accurate, but very good in my experience. GPT-3.5 not good at fact checking at all, it will also hallucinate a lot of incorrect facts. It generates closer to plausible than accurate facts.

dragonwriter 3y ago
> GPT-3.5 not good at fact checking at all, it will also hallucinate a lot of incorrect facts. Even with access to external information through a very basic ReAct implementation GPT-3.5 is fairly decent at fact checking. Without external information to check against, whatever GPT-x is doing isn’t fact-checking. GPT-4 has probably memorized more things because it is a bigger model that has been trained considerably more on a larger training corpus than GPT-3.5, but memorization and fact-checking are different things.
- clnq 3y ago
  It depends on the definition of fact-checking. GPT-4 definitely is able to fact-check most statements more accurately than the human brain could in that same time. I would estimate that upon hearing a fact, I will be able to correctly say whether it’s accurate upon 5 seconds of Googling about 60% of the time. GPT-4 would beat me significantly. If we are talking about investigation-level fact-checking only using primary or academic sources, and spending significant time, effort, and resources to check the facts like some journalists do, then GPT-4 will probably do much worse. But just because we can say “in situation X, it will do better; in situation Y, it will do worse”, we can say that this capability exists. It can be fact-checking. Plug-ins help a lot with that as well. I think with plug-ins, it will meet your definition of fact-checking when asked. Unless it’s a very specific definition.

by clnq 3y ago

dragonwriter 3y ago

> GPT-3.5 not good at fact checking at all, it will also hallucinate a lot of incorrect facts. Even with access to external information through a very basic ReAct implementation GPT-3.5 is fairly decent at fact checking. Without external information to check against, whatever GPT-x is doing isn’t fact-checking. GPT-4 has probably memorized more things because it is a bigger model that has been trained considerably more on a larger training corpus than GPT-3.5, but memorization and fact-checking are different things.

clnq 3y ago
It depends on the definition of fact-checking. GPT-4 definitely is able to fact-check most statements more accurately than the human brain could in that same time. I would estimate that upon hearing a fact, I will be able to correctly say whether it’s accurate upon 5 seconds of Googling about 60% of the time. GPT-4 would beat me significantly. If we are talking about investigation-level fact-checking only using primary or academic sources, and spending significant time, effort, and resources to check the facts like some journalists do, then GPT-4 will probably do much worse. But just because we can say “in situation X, it will do better; in situation Y, it will do worse”, we can say that this capability exists. It can be fact-checking. Plug-ins help a lot with that as well. I think with plug-ins, it will meet your definition of fact-checking when asked. Unless it’s a very specific definition.