3 ms·
Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like rulin
by simonw 3mo ago
Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.
- skorniienko 3mo agoLove it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!
- westoque 3mo agoI don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.
- bulbar 3mo agoIt should not be seen a credible source though. It can be, but that's hard to correctly judge on at scale.
- simonw 3mo agoThe reputation of the writer matters a lot. Recent models seem to be pretty good at evaluating author credibility as well. What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.
- skorniienko 3mo agoAdded this as issue so I don't forget https://github.com/SerhiiKorniienko/bullshit-detector/issues/4 https://github.com/SerhiiKorniienko/bullshit-detector/issues...
- vunderba 3mo agoFor any LLM with web tool usage I’ll often suggest that it pull relevant papers from arXiv, with an expectation that it cite sources. I find that this helps improve accuracy and lets me go deeper into the material if I want to.
- skorniienko 3mo agoHi, ran the experiment with "credible sources". Results we unexpected but not less interesting. Few things worth noting: - same search query ran minutes apart return different sources, sometimes with only 1-2 overlaps. - 16 high-credibility news outlets are rejecting agent's access to a content. (like Reuters, NYT, Guardian), but all .gov, dedicated fact-checkers are working. Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/main/experiments/2026-07-30-credible-sources.md https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...