4 ms·

Yeah, I see that now. Completely useless.

by 4ad 2y ago

simonw 2y ago
Why is that useless? If you want reliable citations, this is an API that can help you implement reliable citations. Asking a model to return useful citations from its model weights with no assistance from external systems isn't how this stuff works.
- 4ad 2y ago
  If I had the stuff to feed into the LLM context, I wouldn't need the LLM to find the stuff for me because I would already have it.
  - dllthomas 2y ago
    "Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced. Certainly it's radically less useful than if it could produce citations to the training set, though.
  - simonw 2y ago
    Sounds to me like you want a search engine, not an LLM.
  - dragonwriter 2y ago
    Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.
  - vunderba 2y ago
    Patently untrue. If I have hundreds of PDF research papers totaling thousands of pages - just because I have them doesn't necessarily mean I know where to search. Having an LLM be able to find related pieces across all of these documents and extrapolate is tremendously useful.
    - 4ad 2y ago
      A thousand pages is something reviewable by hand (albeit slowly) and easily amendable to grep. Anything that trivially fits in computer memory doesn't benefit from approximate search methods. I stand by what I said.
- robwwilliams 2y ago
  Ditto: RAGs are currently way too limited. A RAG built using all of PubMed and updated weekly would be a very different story. I would pay good money for that.
saaaaaam 2y ago
Not useless at all. This is exactly my primary use of the Anthropic api: feed in a source document and ask for specific outputs that include exact citations from the source I provide.

Yeah, I see that now. Completely useless.

by 4ad 2y ago

Yeah, I see that now. Completely useless.

Why is that useless? If you want reliable citations, this is an API that can help you implement reliable citations. Asking a model to return useful citations from its model weights with no assistance from external systems isn't how this stuff works.

4ad 2y ago
If I had the stuff to feed into the LLM context, I wouldn't need the LLM to find the stuff for me because I would already have it.
- dllthomas 2y ago
  "Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced. Certainly it's radically less useful than if it could produce citations to the training set, though.
- simonw 2y ago
  Sounds to me like you want a search engine, not an LLM.
- dragonwriter 2y ago
  Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.
- vunderba 2y ago
  Patently untrue. If I have hundreds of PDF research papers totaling thousands of pages - just because I have them doesn't necessarily mean I know where to search. Having an LLM be able to find related pieces across all of these documents and extrapolate is tremendously useful.
  - 4ad 2y ago
    A thousand pages is something reviewable by hand (albeit slowly) and easily amendable to grep. Anything that trivially fits in computer memory doesn't benefit from approximate search methods. I stand by what I said.
robwwilliams 2y ago
Ditto: RAGs are currently way too limited. A RAG built using all of PubMed and updated weekly would be a very different story. I would pay good money for that.

saaaaaam 2y ago

Not useless at all. This is exactly my primary use of the Anthropic api: feed in a source document and ask for specific outputs that include exact citations from the source I provide.