2 ms·
It's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and
by simonw 2y ago
It's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and if they DID we don't know if that training process might conceivably allow the model to associate IDs with article information.
Given that, I don't trust the models to be able to provide useful citations.
We already know how to get much more reliable citations out of a model: implement them on top of RAG, which this new Claude API can clearly help us do.
- robwwilliams 2y agoIt is definitely not impossible to know. Is there a reason why you or better yet, Anthropic, cannot get empirical FDRs for different areas of research? In my areas, genetics, neuroscience, geroscience, FDR is below 0.2 on articles in PubMed.
- simonw 2y agoAsk Anthropic. I'll be surprised if you can get a straight answer out of them though, like most other labs they are VERY secretive about what actually goes into their training data.
- willy_k 2y agoYou don’t need a straight answer out of anybody. This is an easily verifiable property of the LLM output - are the citations it gives correct, and if so how many?