6 ms·

Ironically, 'source checking' is something AI is quite good at.

by wing-_-nuts 5mo ago

6stringmerc 5mo ago
Citation needed, please
- wing-_-nuts 5mo ago
  Personal experience? You ask it for the name of the paper referenced. You google that paper (for some reason it's not great at going out and acquiring the paper). You then upload the pdf and ask it if the paper supports the assertion if it's not quickly findable via ^F. You go read, ask it clarifying questions about hazard ratios, what they controlled for, etc. AI is quite good when grounded in a source.
flail 5mo ago
There's nuance to that. An LLM is quite capable of suggesting relevant reading, given the context. Especially when the context is broad enough that there's enough training data. "Find me research on code reviews, their size, and quality" would give you more than enough reading. Yet, if you start with a claim, like "Longer PRs mean worse defect detection," the relevant data points fall to few enough for AI to start hallucinating. You get "something, something, PR length, defect detection, IDK, I don't read research papers." Such output is fine as long as the author cares to validate it. Skip the second step, and you might be good if you ask about something generic, like "What's the Slack story?" or "How did Blockbuster go bust?" Ask about some specific details, though, and you're bound to end up with made-up stuff that sounds just about right, while it's actually wrong.
- throw310822 5mo ago
  Checking is different from finding, though. Source checking means just "verify that this information is actually present in that document". Much harder to hallucinate in this case.
  - Kim_Bruning 5mo ago
    A quick smoke check takes just a few minutes. "Follow each link in this document. Read each link's contents against the contents in this document. Create a report: for each link list a working hyperlink, whether it exists, what claim it supports, whether it supports or fails to support it, and why" If it returns a report claiming all correct? That's promising, but human verification is important. You've got a list of hyperlinks, and a list of claims; so you can click each with middle-mouse, Ctrl-F 'till you find the point, and close the tab when you do. If you find any discrepancies ? Your initial prompt was malformed and/or you picked the wrong LLM, the wrong human, or possibly all three. Whatever the way, the results are built on quicksand; you'll need to start over. If no sources are provided? Well now: "If there ain't no sources it never happened." Compare double-entry bookkeeping. It needs to all add up. If you're 1 cent off, that means something is broken. Idem if a single reference is off, it polluted the context. (This works for human-generated and hybrid documents too. Polluted reasoning is polluted reasoning. The process is what counts.)
    - deleted 5mo ago
      [deleted]
    - throw310822 5mo ago
      I would love to do it at scale on many online publications, and publish the results. That would teach 'em.
    - flail 5mo ago
      A quick smoke test, then. Gemini 3, Thinking Mode. The article: https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-physically https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-p... The prompt: literally what you suggested. Gemini: The article focuses on the environmental and human labor costs of scaling Artificial Intelligence, specifically focusing on water usage, electricity, and "ghost work." Which is hilarious, since the article doesn't even mention the words "water" or "electricity." Gemini remains unfazed, reporting the links that are not in the article (some don't exist at all) to make the final ruling: "The Tech Trenches document is highly accurate in its citations." Now, I know. Had I used Claude Code with relevant skills, it would have done better. But would it be good?
kakugawa 5mo ago
Have we forgotten how bad LLMs were at citing sources when they first came out? So, we had to build a lot of structure (harness engineering) and frontier labs had to do specific training to try to compensate for this. So, LLMs are inherently bad at citing sources. A lot of effort has been put in to improve this behavior, but it's compensating for an inherent flaw.
- Kim_Bruning 5mo ago
  Huh? Oh! Were they still treating the LLM as an "oracle box"/online chatbot at the time? (as opposed to a more agentic workflow?) If they weren't, ignore I said the following, and please tell me what else was going wrong (and with what models and harnesses!). Models weights are like Wikipedia. A nice starting point, but should never be referenced directly. You need to have your agent actually go out onto the internet and do the research. Now the actual references will be in your agent's actual Context (memory), so then it'd at least be rather more surprising if they don't cite correctly. I do realize there's still corner cases even in the best setups though; So a final crosscheck sweep is never not a good idea.
oulipo2 5mo ago
Judging by the number of scientific papers that have been outed as AI-generated, precisely because it hallucinated sources, it's not
1970-01-01 5mo ago
I disagree. It is a bullshit machine all the way to the core. LLMs in my world fail to cite full sources and consistently conclude with guesses as facts. It does this much more than an average journalist or reporter would. Only when you double-check it will it then apologize and correct itself.
Thrymr 5mo ago
I mostly disagree with this. You can request sources, you can ask it to check, but no LLM I have used can do this correctly more than 50-75% of the time, and some of the major models are extremely bad at this: giving broken links 90% of the time, incapable of giving actual links rather than search engine links, etc. Constant supervision and repetition of requests can sometimes get results, but it is exhausting. The "sources" it finds are often Reddit posts or other questionable secondary or tertiary sources, not actual original sources.

Ironically, 'source checking' is something AI is quite good at.

6stringmerc 5mo ago

Citation needed, please

wing-_-nuts 5mo ago
Personal experience? You ask it for the name of the paper referenced. You google that paper (for some reason it's not great at going out and acquiring the paper). You then upload the pdf and ask it if the paper supports the assertion if it's not quickly findable via ^F. You go read, ask it clarifying questions about hazard ratios, what they controlled for, etc. AI is quite good when grounded in a source.

flail 5mo ago

There's nuance to that. An LLM is quite capable of suggesting relevant reading, given the context. Especially when the context is broad enough that there's enough training data. "Find me research on code reviews, their size, and quality" would give you more than enough reading. Yet, if you start with a claim, like "Longer PRs mean worse defect detection," the relevant data points fall to few enough for AI to start hallucinating. You get "something, something, PR length, defect detection, IDK, I don't read research papers." Such output is fine as long as the author cares to validate it. Skip the second step, and you might be good if you ask about something generic, like "What's the Slack story?" or "How did Blockbuster go bust?" Ask about some specific details, though, and you're bound to end up with made-up stuff that sounds just about right, while it's actually wrong.

throw310822 5mo ago
Checking is different from finding, though. Source checking means just "verify that this information is actually present in that document". Much harder to hallucinate in this case.
- Kim_Bruning 5mo ago
  A quick smoke check takes just a few minutes. "Follow each link in this document. Read each link's contents against the contents in this document. Create a report: for each link list a working hyperlink, whether it exists, what claim it supports, whether it supports or fails to support it, and why" If it returns a report claiming all correct? That's promising, but human verification is important. You've got a list of hyperlinks, and a list of claims; so you can click each with middle-mouse, Ctrl-F 'till you find the point, and close the tab when you do. If you find any discrepancies ? Your initial prompt was malformed and/or you picked the wrong LLM, the wrong human, or possibly all three. Whatever the way, the results are built on quicksand; you'll need to start over. If no sources are provided? Well now: "If there ain't no sources it never happened." Compare double-entry bookkeeping. It needs to all add up. If you're 1 cent off, that means something is broken. Idem if a single reference is off, it polluted the context. (This works for human-generated and hybrid documents too. Polluted reasoning is polluted reasoning. The process is what counts.)
  - deleted 5mo ago
    [deleted]
  - throw310822 5mo ago
    I would love to do it at scale on many online publications, and publish the results. That would teach 'em.
  - flail 5mo ago
    A quick smoke test, then. Gemini 3, Thinking Mode. The article: https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-physically https://techtrenches.dev/p/the-human-cost-of-10x-how-ai-is-p... The prompt: literally what you suggested. Gemini: The article focuses on the environmental and human labor costs of scaling Artificial Intelligence, specifically focusing on water usage, electricity, and "ghost work." Which is hilarious, since the article doesn't even mention the words "water" or "electricity." Gemini remains unfazed, reporting the links that are not in the article (some don't exist at all) to make the final ruling: "The Tech Trenches document is highly accurate in its citations." Now, I know. Had I used Claude Code with relevant skills, it would have done better. But would it be good?

kakugawa 5mo ago

Have we forgotten how bad LLMs were at citing sources when they first came out? So, we had to build a lot of structure (harness engineering) and frontier labs had to do specific training to try to compensate for this. So, LLMs are inherently bad at citing sources. A lot of effort has been put in to improve this behavior, but it's compensating for an inherent flaw.

Kim_Bruning 5mo ago
Huh? Oh! Were they still treating the LLM as an "oracle box"/online chatbot at the time? (as opposed to a more agentic workflow?) If they weren't, ignore I said the following, and please tell me what else was going wrong (and with what models and harnesses!). Models weights are like Wikipedia. A nice starting point, but should never be referenced directly. You need to have your agent actually go out onto the internet and do the research. Now the actual references will be in your agent's actual Context (memory), so then it'd at least be rather more surprising if they don't cite correctly. I do realize there's still corner cases even in the best setups though; So a final crosscheck sweep is never not a good idea.

oulipo2 5mo ago

Judging by the number of scientific papers that have been outed as AI-generated, precisely because it hallucinated sources, it's not

1970-01-01 5mo ago

I disagree. It is a bullshit machine all the way to the core. LLMs in my world fail to cite full sources and consistently conclude with guesses as facts. It does this much more than an average journalist or reporter would. Only when you double-check it will it then apologize and correct itself.

Thrymr 5mo ago

I mostly disagree with this. You can request sources, you can ask it to check, but no LLM I have used can do this correctly more than 50-75% of the time, and some of the major models are extremely bad at this: giving broken links 90% of the time, incapable of giving actual links rather than search engine links, etc. Constant supervision and repetition of requests can sometimes get results, but it is exhausting. The "sources" it finds are often Reddit posts or other questionable secondary or tertiary sources, not actual original sources.