4 ms·

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giv

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a legal or medical opinion supported by research.

freedomben 3mo ago
For standard responses you're not wrong, but increasingly there are a lot of LLMS that are essentially doing RAG against search results. For example this is I believe how Kagi works, and Google AI overviews. I have increasingly seen Claude and chat GPT also doing the same thing where instead of answering a question from the knowledge Bank they will do a web search and cite the responses that were used.
SecretDreams 3mo ago
Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.
- pfdietz 3mo ago
  It's hilariously common online as well. Bullshitter makes a claim, someone calls them on the BS, they put up a link, and (lo and behold) the link when examined does nothing to defend the claim. Often the link directly contradicts the claim.
  - SecretDreams 3mo ago
    Yep..the online link posters hit close to home. Big reason I hardly even engage in discourse anymore. Got tired of verifying links that don't have the claimed content in them.
- wwweston 3mo ago
  with the advantage that you can discover this by reading citations.
  - SecretDreams 3mo ago
    Yes, but it takes a lot of time to find a lie and very little time to conceive a lie. It's a losing battle as the percentage of honest citations drops (and it doesn't need to be a big drop to have a big impact.. think if the reliability of planes dropped 1%).
anon373839 3mo ago
This is true, but good LLMs (even local, consumer-sized ones these days) can do this much better than … whatever it is that Google’s AI overviews use. Think about your coding harness: the model reads a bunch of files into the context, and it generally doesn’t forget/hallucinate which lines came from which file.
- captainbland 3mo ago
  I'd be surprised if it were anything other than just an LLM only equivalent of Gemma some 4B param model, they can't be spending much money on it because each query essentially needs to be cheaper than the ad revenue per search.
  - anon373839 3mo ago
    Could they even afford 4B dense parameters touching every token? That seems like a LOT of compute compared to generating the classic Google SERP.
chollida1 3mo ago
> It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. I mean, its not. Lots of LLM's focused on finance do this already. Bloomberg's own ASKB produces results and provides links back to the source documents or urls that it references so people who care about correctness can verify the results.
jefftk 3mo ago
You can do it the same way human does: recall a fact from memory, and then search to identify a citation. The citation isn't "here's why I think this" but instead "here's where you can verify this".
- knollimar 3mo ago
  I don't understand why verifying isn't in the loop. If you gave an LLM: "is this statement consistent with this RAG data?" it would do well. It feels lazy that this isn't built into the harness in some adversarial citation review checkbox. Also the questions LLMs ask are so inhuman. It's like it's some contrarian trying to win an argument over getting an answer.
amelius 3mo ago
Then the least they can do is cite ALL their sources. At least somewhere on their website.
- brookst 3mo ago
  So an HTML page with a list of essentially every page on every site on the internet? What do you think, maybe a trillion URLs?
  - amelius 3mo ago
    Yes. Whatever makes their lawyers nervous.
WarmWash 3mo ago
This is why it's dumb when inferencing with a model to ask "...and cite all sources". Total waste of time and likely to make the response worse. BUT On models with web search, they can scan the sources in context, and those are pretty good at correct citations. However it's still a "trust, but verify" situation.
ordersofmag 3mo ago
99% of AI users these days aren't using a 'raw' LLM. They are interacting with a harness that includes tools to do search of the live (or recently crawled) web. And so the output users actually see could absolutely include correct citation of sources. So the 'architecturally impossible' bit may be technically correct but it is not practically relevant. Now whether those harnesses do a good job of orchestrating LLM output to get accurate citations (and whether they are transparent about their process) is another thing entirely. But if you're contemplating 'what LLM's can do' and aren't taking into account the harness and tooling ecosystem they are embedded in then you're missing the point.
ssl-3 3mo ago
> It is perfectly common to find that a "source" does not contain the statements. It is commonly this way right now, but it doesn't have to stay this way forever. It is also common, in my usage at least, to iteratively brow-beat the bot into paring its statements down to those that which are supportable by its sources. Doing so just takes repetition, and that repetition takes time and burns more tokens. With the present state of things, the prompts to get moving on this and to guide the ultimate response into something that is verifiably supportable by outside sources can usually be simple and largely generic. They're easy enough prompts that a subagent can produce them. (I've done it myself with Codex subagents and it worked very well, aside from the unsustainable burn rate that did not fit my budget.)
razodactyl 3mo ago
Are you sure? Because this sounds like GPT3-era understandings of LLM-isms

freedomben 3mo ago

For standard responses you're not wrong, but increasingly there are a lot of LLMS that are essentially doing RAG against search results. For example this is I believe how Kagi works, and Google AI overviews. I have increasingly seen Claude and chat GPT also doing the same thing where instead of answering a question from the knowledge Bank they will do a web search and cite the responses that were used.

SecretDreams 3mo ago

Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.

pfdietz 3mo ago
It's hilariously common online as well. Bullshitter makes a claim, someone calls them on the BS, they put up a link, and (lo and behold) the link when examined does nothing to defend the claim. Often the link directly contradicts the claim.
- SecretDreams 3mo ago
  Yep..the online link posters hit close to home. Big reason I hardly even engage in discourse anymore. Got tired of verifying links that don't have the claimed content in them.
wwweston 3mo ago
with the advantage that you can discover this by reading citations.
- SecretDreams 3mo ago
  Yes, but it takes a lot of time to find a lie and very little time to conceive a lie. It's a losing battle as the percentage of honest citations drops (and it doesn't need to be a big drop to have a big impact.. think if the reliability of planes dropped 1%).

anon373839 3mo ago

This is true, but good LLMs (even local, consumer-sized ones these days) can do this much better than … whatever it is that Google’s AI overviews use. Think about your coding harness: the model reads a bunch of files into the context, and it generally doesn’t forget/hallucinate which lines came from which file.

captainbland 3mo ago
I'd be surprised if it were anything other than just an LLM only equivalent of Gemma some 4B param model, they can't be spending much money on it because each query essentially needs to be cheaper than the ad revenue per search.
- anon373839 3mo ago
  Could they even afford 4B dense parameters touching every token? That seems like a LOT of compute compared to generating the classic Google SERP.

chollida1 3mo ago

> It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. I mean, its not. Lots of LLM's focused on finance do this already. Bloomberg's own ASKB produces results and provides links back to the source documents or urls that it references so people who care about correctness can verify the results.

jefftk 3mo ago

You can do it the same way human does: recall a fact from memory, and then search to identify a citation. The citation isn't "here's why I think this" but instead "here's where you can verify this".

knollimar 3mo ago
I don't understand why verifying isn't in the loop. If you gave an LLM: "is this statement consistent with this RAG data?" it would do well. It feels lazy that this isn't built into the harness in some adversarial citation review checkbox. Also the questions LLMs ask are so inhuman. It's like it's some contrarian trying to win an argument over getting an answer.

amelius 3mo ago

Then the least they can do is cite ALL their sources. At least somewhere on their website.

brookst 3mo ago
So an HTML page with a list of essentially every page on every site on the internet? What do you think, maybe a trillion URLs?
- amelius 3mo ago
  Yes. Whatever makes their lawyers nervous.

WarmWash 3mo ago

This is why it's dumb when inferencing with a model to ask "...and cite all sources". Total waste of time and likely to make the response worse. BUT On models with web search, they can scan the sources in context, and those are pretty good at correct citations. However it's still a "trust, but verify" situation.

ordersofmag 3mo ago

99% of AI users these days aren't using a 'raw' LLM. They are interacting with a harness that includes tools to do search of the live (or recently crawled) web. And so the output users actually see could absolutely include correct citation of sources. So the 'architecturally impossible' bit may be technically correct but it is not practically relevant. Now whether those harnesses do a good job of orchestrating LLM output to get accurate citations (and whether they are transparent about their process) is another thing entirely. But if you're contemplating 'what LLM's can do' and aren't taking into account the harness and tooling ecosystem they are embedded in then you're missing the point.

ssl-3 3mo ago

> It is perfectly common to find that a "source" does not contain the statements. It is commonly this way right now, but it doesn't have to stay this way forever. It is also common, in my usage at least, to iteratively brow-beat the bot into paring its statements down to those that which are supportable by its sources. Doing so just takes repetition, and that repetition takes time and burns more tokens. With the present state of things, the prompts to get moving on this and to guide the ultimate response into something that is verifiably supportable by outside sources can usually be simple and largely generic. They're easy enough prompts that a subagent can produce them. (I've done it myself with Codex subagents and it worked very well, aside from the unsustainable burn rate that did not fit my budget.)

razodactyl 3mo ago

Are you sure? Because this sounds like GPT3-era understandings of LLM-isms