3 ms·

Have you tested this extensively yourself? I have been very surprised by success rates in my own “not famous” papers. I asked Claude to provide full citation to

Have you tested this extensively yourself? I have been very surprised by success rates in my own “not famous” papers. I asked Claude to provide full citation to ten papers by Robert W Williams at the University of Tennessee in biomedical research. Nine of ten were perfect, down to page numbers. One if ten was a complete construct but highly plausible. An FDR of 0.1 is damn impressive. Test yourself. Here was my reference data set: https://scholar.google.com/citations?user=OYJMYwIAAAAJ&hl=en&oi=ao https://scholar.google.com/citations?user=OYJMYwIAAAAJ&hl=en... Really curious what the range of FDRs is at different levels of accuracy for different fields.

BoorishBears 2y ago
It's fundamentally not something anyone building a real application can rely on. You're essentially gambling on where in the embedding space your end users are going to query: you might get lucky or you might not. You're also relying on functionality they're actively trying to degrade during post-training (repeating training data text verbatim). Some LLM providers will even actively filter it: https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python#recitation-issue https://ai.google.dev/gemini-api/docs/troubleshooting?lang=p...
- diggan 2y ago
  > It's fundamentally not something anyone building a real application can rely on. I don't think parent is arguing for just connecting an LLM with that prompt to random end-users, they're talking about using that prompt for their own work, which I'd assume goes through the usual process of being verified and so on.
- robwwilliams 2y ago
  In some cases, sure, you are absolutely right. But in many cases “application you can rely on” depends on the context, the field, and user expectations. I “rely” on weather report apps but expect inaccuracies. I rely on medical tests results but expect FDRs of about .05. In contrast, I expect financial apps to have very low FDRs. I use Claude as a tool to explore research areas and topics. Here is an example of a recent prompt: “Please provide references to papers that have reported swelling of brain tissue (increase in volume) in neurodegenerative disease such as Alzheimer’s using MRI methods.” After a back-and-forth I got just what I needed, and more easily than a PubMed query—-two references that demonstrate early-stage volume increases in some parts of the brain of adults with presenilin 1 mutations.
  - simonw 2y ago
    It's important to understand what's going on here. Claude isn't providing references because it has a copy of the papers and knows their URLs. It's instead providing references because enough of the OTHER stuff in the training data (which is entirely undocumented but we can assume includes a scrape of the web, blog posts, Reddit, other papers etc) talked about those papers in a way that provided context on the paper and enough of a title/URL that Claude could usefully return that. By its nature then you'll only ever find references to VERY popular papers that way. That might be what you want! But you shouldn't expect Claude to be able to dig into the lesser known research around a topic based exclusively on its model weights. For that you'd be much better off with a system like Gemini Deep Research: https://blog.google/products/gemini/google-gemini-deep-research/ https://blog.google/products/gemini/google-gemini-deep-resea...
    - robwwilliams 2y ago
      Not consistent with my empirical tests. I asked Claude for references on the demographics of wild rodent populations—-not exactly a hot topic, and all four references were spot on, including two not even listed in PubMed.
simonw 2y ago
I'll be honest, I was surprised at how well Claude was able to describe the papers you linked to there when I prompted it directly about them. I may have to update my mental model of quite how good Claude's recall can be with respect to academic papers in those areas.
vunderba 2y ago
Maybe, but there's a big difference between a known 10% failure rate versus an unknown 10% failure rate. The former allows me to instantly reject the failures and retain the good responses. The latter requires me to go through a manual check for every response.

BoorishBears 2y ago

It's fundamentally not something anyone building a real application can rely on. You're essentially gambling on where in the embedding space your end users are going to query: you might get lucky or you might not. You're also relying on functionality they're actively trying to degrade during post-training (repeating training data text verbatim). Some LLM providers will even actively filter it: https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python#recitation-issue https://ai.google.dev/gemini-api/docs/troubleshooting?lang=p...

diggan 2y ago
> It's fundamentally not something anyone building a real application can rely on. I don't think parent is arguing for just connecting an LLM with that prompt to random end-users, they're talking about using that prompt for their own work, which I'd assume goes through the usual process of being verified and so on.
robwwilliams 2y ago
In some cases, sure, you are absolutely right. But in many cases “application you can rely on” depends on the context, the field, and user expectations. I “rely” on weather report apps but expect inaccuracies. I rely on medical tests results but expect FDRs of about .05. In contrast, I expect financial apps to have very low FDRs. I use Claude as a tool to explore research areas and topics. Here is an example of a recent prompt: “Please provide references to papers that have reported swelling of brain tissue (increase in volume) in neurodegenerative disease such as Alzheimer’s using MRI methods.” After a back-and-forth I got just what I needed, and more easily than a PubMed query—-two references that demonstrate early-stage volume increases in some parts of the brain of adults with presenilin 1 mutations.
- simonw 2y ago
  It's important to understand what's going on here. Claude isn't providing references because it has a copy of the papers and knows their URLs. It's instead providing references because enough of the OTHER stuff in the training data (which is entirely undocumented but we can assume includes a scrape of the web, blog posts, Reddit, other papers etc) talked about those papers in a way that provided context on the paper and enough of a title/URL that Claude could usefully return that. By its nature then you'll only ever find references to VERY popular papers that way. That might be what you want! But you shouldn't expect Claude to be able to dig into the lesser known research around a topic based exclusively on its model weights. For that you'd be much better off with a system like Gemini Deep Research: https://blog.google/products/gemini/google-gemini-deep-research/ https://blog.google/products/gemini/google-gemini-deep-resea...
  - robwwilliams 2y ago
    Not consistent with my empirical tests. I asked Claude for references on the demographics of wild rodent populations—-not exactly a hot topic, and all four references were spot on, including two not even listed in PubMed.

simonw 2y ago

I'll be honest, I was surprised at how well Claude was able to describe the papers you linked to there when I prompted it directly about them. I may have to update my mental model of quite how good Claude's recall can be with respect to academic papers in those areas.

vunderba 2y ago

Maybe, but there's a big difference between a known 10% failure rate versus an unknown 10% failure rate. The former allows me to instantly reject the failures and retain the good responses. The latter requires me to go through a manual check for every response.