3 ms·

What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being t

What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?

skorniienko 3mo ago
There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.
- gutechh 3mo ago
  Maybe you read them already, but wikipedia has a bunch of interesting pages about sourcing and truth like: https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_truth https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_t... and https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources
  - skorniienko 3mo ago
    I'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something. Will read reliable-sources page properly.
- desktopentree 3mo ago
  [flagged]
simonw 3mo ago
Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.
- skorniienko 3mo ago
  Love it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!
- westoque 3mo ago
  I don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.
  - bulbar 3mo ago
    It should not be seen a credible source though. It can be, but that's hard to correctly judge on at scale.
  - simonw 3mo ago
    The reputation of the writer matters a lot. Recent models seem to be pretty good at evaluating author credibility as well. What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.
- skorniienko 3mo ago
  Added this as issue so I don't forget https://github.com/SerhiiKorniienko/bullshit-detector/issues/4 https://github.com/SerhiiKorniienko/bullshit-detector/issues...
- vunderba 3mo ago
  For any LLM with web tool usage I’ll often suggest that it pull relevant papers from arXiv, with an expectation that it cite sources. I find that this helps improve accuracy and lets me go deeper into the material if I want to.
- skorniienko 3mo ago
  Hi, ran the experiment with "credible sources". Results we unexpected but not less interesting. Few things worth noting: - same search query ran minutes apart return different sources, sometimes with only 1-2 overlaps. - 16 high-credibility news outlets are rejecting agent's access to a content. (like Reuters, NYT, Guardian), but all .gov, dedicated fact-checkers are working. Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/main/experiments/2026-07-30-credible-sources.md https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...
autonomousErwin 3mo ago
I could imagine running multiple different AIs seeing if they agree if something is true. I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true. So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.
- mtweak 3mo ago
  This is extremely problematic. Much of news is highly syndicated, so what looks like 10 credible sources are actually just 2. I think the best you can aim for is empirical sources: receipts, videos, photo evidence, public disclosures directly at the source. Archiving is ok if disclosed. I don't think LLMs can discern truth when the majority of news sources are incentivized primarily for views.
  - skinfaxi 3mo ago
    You may have overlooked this part of the parent's comment which addresses your syndication issue: > Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true. Outfits that blindly parrot talking points would not be at odds and thus not be good candidates.
  - hunterpayne 3mo ago
    It also doesn't help that many news sources will knowingly fudge the truth. There are topics about which they are constant sources of misinformation. Seems like depending on the topic (anything science related) you wouldn't want to use news sources at all. Other topics, you probably only want to use news sources. And arranging all of this weighting of sources is going to be difficult and controversial as well. Doesn't mean it can't be done. Doesn't mean it wouldn't be valuable. But there is a lot to it and most of it isn't so much about technology or math as it is about understanding who should and shouldn't be considered authoritative about what topics. Sounds challenging to me...

skorniienko 3mo ago

There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.

gutechh 3mo ago
Maybe you read them already, but wikipedia has a bunch of interesting pages about sourcing and truth like: https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_truth https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_t... and https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources
- skorniienko 3mo ago
  I'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something. Will read reliable-sources page properly.
desktopentree 3mo ago
[flagged]

simonw 3mo ago

Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.

skorniienko 3mo ago
Love it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!
westoque 3mo ago
I don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.
- bulbar 3mo ago
  It should not be seen a credible source though. It can be, but that's hard to correctly judge on at scale.
- simonw 3mo ago
  The reputation of the writer matters a lot. Recent models seem to be pretty good at evaluating author credibility as well. What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.
skorniienko 3mo ago
Added this as issue so I don't forget https://github.com/SerhiiKorniienko/bullshit-detector/issues/4 https://github.com/SerhiiKorniienko/bullshit-detector/issues...
vunderba 3mo ago
For any LLM with web tool usage I’ll often suggest that it pull relevant papers from arXiv, with an expectation that it cite sources. I find that this helps improve accuracy and lets me go deeper into the material if I want to.
skorniienko 3mo ago
Hi, ran the experiment with "credible sources". Results we unexpected but not less interesting. Few things worth noting: - same search query ran minutes apart return different sources, sometimes with only 1-2 overlaps. - 16 high-credibility news outlets are rejecting agent's access to a content. (like Reuters, NYT, Guardian), but all .gov, dedicated fact-checkers are working. Detailed report, if you are interested: https://github.com/SerhiiKorniienko/bullshit-detector/blob/main/experiments/2026-07-30-credible-sources.md https://github.com/SerhiiKorniienko/bullshit-detector/blob/m...

autonomousErwin 3mo ago

I could imagine running multiple different AIs seeing if they agree if something is true. I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true. So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.

mtweak 3mo ago
This is extremely problematic. Much of news is highly syndicated, so what looks like 10 credible sources are actually just 2. I think the best you can aim for is empirical sources: receipts, videos, photo evidence, public disclosures directly at the source. Archiving is ok if disclosed. I don't think LLMs can discern truth when the majority of news sources are incentivized primarily for views.
- skinfaxi 3mo ago
  You may have overlooked this part of the parent's comment which addresses your syndication issue: > Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true. Outfits that blindly parrot talking points would not be at odds and thus not be good candidates.
- hunterpayne 3mo ago
  It also doesn't help that many news sources will knowingly fudge the truth. There are topics about which they are constant sources of misinformation. Seems like depending on the topic (anything science related) you wouldn't want to use news sources at all. Other topics, you probably only want to use news sources. And arranging all of this weighting of sources is going to be difficult and controversial as well. Doesn't mean it can't be done. Doesn't mean it wouldn't be valuable. But there is a lot to it and most of it isn't so much about technology or math as it is about understanding who should and shouldn't be considered authoritative about what topics. Sounds challenging to me...