3 ms·
Saying to Claude "Always suggest good texts with full references and even PubMed IDs" is asking it to do the impossible: it doesn't have the ability to identify
by simonw 2y ago
Saying to Claude "Always suggest good texts with full references and even PubMed IDs" is asking it to do the impossible: it doesn't have the ability to identify which information in its knowledge comes from which PubMed ID reference sources, so it's right that it refuses to do that even when you tell it to.
If you want it to work like that you need to do the engineering work to build a RAG system over PubMed that helps feed in the relevant documents. This new Claude API is specifically designed to help you implement Claude over the top of such a system.
- willy_k 2y agoWhat’s the actual hit rate? AFAIU it’s plausible that an article title and PubMed ID could be encoded in the models weights if they were trained on.
- simonw 2y agoIt's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and if they DID we don't know if that training process might conceivably allow the model to associate IDs with article information. Given that, I don't trust the models to be able to provide useful citations. We already know how to get much more reliable citations out of a model: implement them on top of RAG, which this new Claude API can clearly help us do.
- robwwilliams 2y agoIt is definitely not impossible to know. Is there a reason why you or better yet, Anthropic, cannot get empirical FDRs for different areas of research? In my areas, genetics, neuroscience, geroscience, FDR is below 0.2 on articles in PubMed.
- simonw 2y agoAsk Anthropic. I'll be surprised if you can get a straight answer out of them though, like most other labs they are VERY secretive about what actually goes into their training data.
- willy_k 2y agoYou don’t need a straight answer out of anybody. This is an easily verifiable property of the LLM output - are the citations it gives correct, and if so how many?
- robwwilliams 2y agoFor me the true positive rate is at least 80%.
- se4u 2y agoYeah I don't understand why the whole thread has been so hostile against a very reasonable/useful observation that you made. If there was a way to prompt commenters to be less snarky on HN that'd be a vast improvement.
- robwwilliams 2y agoHave you tested this extensively yourself? I have been very surprised by success rates in my own “not famous” papers. I asked Claude to provide full citation to ten papers by Robert W Williams at the University of Tennessee in biomedical research. Nine of ten were perfect, down to page numbers. One if ten was a complete construct but highly plausible. An FDR of 0.1 is damn impressive. Test yourself. Here was my reference data set: https://scholar.google.com/citations?user=OYJMYwIAAAAJ&hl=en&oi=ao https://scholar.google.com/citations?user=OYJMYwIAAAAJ&hl=en... Really curious what the range of FDRs is at different levels of accuracy for different fields.
- BoorishBears 2y agoIt's fundamentally not something anyone building a real application can rely on. You're essentially gambling on where in the embedding space your end users are going to query: you might get lucky or you might not. You're also relying on functionality they're actively trying to degrade during post-training (repeating training data text verbatim). Some LLM providers will even actively filter it: https://ai.google.dev/gemini-api/docs/troubleshooting?lang=python#recitation-issue https://ai.google.dev/gemini-api/docs/troubleshooting?lang=p...
- diggan 2y ago> It's fundamentally not something anyone building a real application can rely on. I don't think parent is arguing for just connecting an LLM with that prompt to random end-users, they're talking about using that prompt for their own work, which I'd assume goes through the usual process of being verified and so on.
- robwwilliams 2y agoIn some cases, sure, you are absolutely right. But in many cases “application you can rely on” depends on the context, the field, and user expectations. I “rely” on weather report apps but expect inaccuracies. I rely on medical tests results but expect FDRs of about .05. In contrast, I expect financial apps to have very low FDRs. I use Claude as a tool to explore research areas and topics. Here is an example of a recent prompt: “Please provide references to papers that have reported swelling of brain tissue (increase in volume) in neurodegenerative disease such as Alzheimer’s using MRI methods.” After a back-and-forth I got just what I needed, and more easily than a PubMed query—-two references that demonstrate early-stage volume increases in some parts of the brain of adults with presenilin 1 mutations.
- no_wizard 2y agoFurther proof AI is t actually AI and we continue to move the goalpost on the proper definition of the term. It’s all still machine learning like always and it has the same limitations.