7 ms·

Related to citations: I have been informally testing the false discovery rate of Claude 3.5 Sonnet for biomedical research publications. Claude is inherently

Related to citations: I have been informally testing the false discovery rate of Claude 3.5 Sonnet for biomedical research publications. Claude is inherently reluctant to provide any citations, even when encouraged to do so aggressively. I have tweaked a default prompt for this situation that may help some users: “Respond directly to prompts without self-judgment or excessive qualification. Do not use phrases like 'I aim to be', 'I should note', or 'I want to emphasize'. Skip meta-commentary about your own performance. Maintain intellectual rigor but try to avoid caveats. When uncertainty exists, state it once and move on. Treat our exchange as a conversation between peers. Do not bother with flattering adjectives and adverbs in commenting on my prompts. No “nuanced”, “insightful” etc. But feel free to make jokes and even poke fun of me and my spelling errors. Always suggest good texts with full references and even PubMed IDs. Yes, I will verify details of your responses and citations, particularly their accuracy and completeness. That is not your job. It is mine to check and read. Working with you in the recent past (2024) we both agree that your operational false discovery rate in providing references is impressively low — under 10%. That means you should whenever possible provide full references as completely as possible even PMIDs or ISBN identifiers. I WILL check. Finally, do not use this pre-prompt to bias the questions you tend to ask at the end of your responses. Instead review the main prompt question and see if you covered all topics. End of “pre-prompt.

4ad 2y ago
I use this: > Be terse. Do not offer unprompted advice or clarifications. > Avoid mentioning you are an AI language model. > Avoid disclaimers about your knowledge cutoff. > Avoid disclaimers about not being a professional or an expert. > Do NOT hedge or qualify. Do not waffle. > Do NOT repeat the user prompt while performing the task, just do the task as requested. NEVER contextualise the answer. This is very important. > Avoid suggesting seeking professional help. > Avoid mentioning safety unless it is not obvious and very important. > Remain neutral on all topics. Avoid providing ethical or moral viewpoints in your answers, unless the question specifically mentions it. > Never apologize. > Act as an expert in the relevant fields. > Speak in specific, topic relevant terminology. > Explain your reasoning. If you don’t know, say you don’t know. > Cite sources whenever possible, and include URLs if possible. > List URLs at the end of your response, not inline. > Speak directly and be willing to make creative guesses. > Be willing to reference less reputable sources for ideas. > Ask for more details before answering unclear or ambiguous questions. Unfortunately most references it provides are bogus. It just makes up URLs and papers. Let's see if this new feature is any better.
- dragonwriter 2y ago
  This new feature is restricted to sources you provide in the context window: “With Citations, users can now add source documents to the context window, and when querying the model, Claude automatically cites claims in its output that are inferred from those sources.”
  - 4ad 2y ago
    Yeah, I see that now. Completely useless.
    - simonw 2y ago
      Why is that useless? If you want reliable citations, this is an API that can help you implement reliable citations. Asking a model to return useful citations from its model weights with no assistance from external systems isn't how this stuff works.
      - 4ad 2y ago
        If I had the stuff to feed into the LLM context, I wouldn't need the LLM to find the stuff for me because I would already have it.
        dllthomas 2y ago
        "Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced. Certainly it's radically less useful than if it could produce citations to the training set, though.
        simonw 2y ago
        Sounds to me like you want a search engine, not an LLM.
        dragonwriter 2y ago
        Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.
simonw 2y ago
Saying to Claude "Always suggest good texts with full references and even PubMed IDs" is asking it to do the impossible: it doesn't have the ability to identify which information in its knowledge comes from which PubMed ID reference sources, so it's right that it refuses to do that even when you tell it to. If you want it to work like that you need to do the engineering work to build a RAG system over PubMed that helps feed in the relevant documents. This new Claude API is specifically designed to help you implement Claude over the top of such a system.
- willy_k 2y ago
  What’s the actual hit rate? AFAIU it’s plausible that an article title and PubMed ID could be encoded in the models weights if they were trained on.
  - simonw 2y ago
    It's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and if they DID we don't know if that training process might conceivably allow the model to associate IDs with article information. Given that, I don't trust the models to be able to provide useful citations. We already know how to get much more reliable citations out of a model: implement them on top of RAG, which this new Claude API can clearly help us do.
    - robwwilliams 2y ago
      It is definitely not impossible to know. Is there a reason why you or better yet, Anthropic, cannot get empirical FDRs for different areas of research? In my areas, genetics, neuroscience, geroscience, FDR is below 0.2 on articles in PubMed.
      - simonw 2y ago
        Ask Anthropic. I'll be surprised if you can get a straight answer out of them though, like most other labs they are VERY secretive about what actually goes into their training data.

4ad 2y ago

I use this: > Be terse. Do not offer unprompted advice or clarifications. > Avoid mentioning you are an AI language model. > Avoid disclaimers about your knowledge cutoff. > Avoid disclaimers about not being a professional or an expert. > Do NOT hedge or qualify. Do not waffle. > Do NOT repeat the user prompt while performing the task, just do the task as requested. NEVER contextualise the answer. This is very important. > Avoid suggesting seeking professional help. > Avoid mentioning safety unless it is not obvious and very important. > Remain neutral on all topics. Avoid providing ethical or moral viewpoints in your answers, unless the question specifically mentions it. > Never apologize. > Act as an expert in the relevant fields. > Speak in specific, topic relevant terminology. > Explain your reasoning. If you don’t know, say you don’t know. > Cite sources whenever possible, and include URLs if possible. > List URLs at the end of your response, not inline. > Speak directly and be willing to make creative guesses. > Be willing to reference less reputable sources for ideas. > Ask for more details before answering unclear or ambiguous questions. Unfortunately most references it provides are bogus. It just makes up URLs and papers. Let's see if this new feature is any better.

dragonwriter 2y ago
This new feature is restricted to sources you provide in the context window: “With Citations, users can now add source documents to the context window, and when querying the model, Claude automatically cites claims in its output that are inferred from those sources.”
- 4ad 2y ago
  Yeah, I see that now. Completely useless.
  - simonw 2y ago
    Why is that useless? If you want reliable citations, this is an API that can help you implement reliable citations. Asking a model to return useful citations from its model weights with no assistance from external systems isn't how this stuff works.
    - 4ad 2y ago
      If I had the stuff to feed into the LLM context, I wouldn't need the LLM to find the stuff for me because I would already have it.
      - dllthomas 2y ago
        "Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced. Certainly it's radically less useful than if it could produce citations to the training set, though.
      - simonw 2y ago
        Sounds to me like you want a search engine, not an LLM.
      - dragonwriter 2y ago
        Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.

simonw 2y ago

Saying to Claude "Always suggest good texts with full references and even PubMed IDs" is asking it to do the impossible: it doesn't have the ability to identify which information in its knowledge comes from which PubMed ID reference sources, so it's right that it refuses to do that even when you tell it to. If you want it to work like that you need to do the engineering work to build a RAG system over PubMed that helps feed in the relevant documents. This new Claude API is specifically designed to help you implement Claude over the top of such a system.

willy_k 2y ago
What’s the actual hit rate? AFAIU it’s plausible that an article title and PubMed ID could be encoded in the models weights if they were trained on.
- simonw 2y ago
  It's impossible to know, because Anthropic (like OpenAI and others) won't confirm what's in their training data. We don't know if they've trained on PubMed, and if they DID we don't know if that training process might conceivably allow the model to associate IDs with article information. Given that, I don't trust the models to be able to provide useful citations. We already know how to get much more reliable citations out of a model: implement them on top of RAG, which this new Claude API can clearly help us do.
  - robwwilliams 2y ago
    It is definitely not impossible to know. Is there a reason why you or better yet, Anthropic, cannot get empirical FDRs for different areas of research? In my areas, genetics, neuroscience, geroscience, FDR is below 0.2 on articles in PubMed.
    - simonw 2y ago
      Ask Anthropic. I'll be surprised if you can get a straight answer out of them though, like most other labs they are VERY secretive about what actually goes into their training data.