2 ms·

The problem is that this is still just retrieval and mechanical. In RAG, you split a PDF into small chunks, but this is way different from how humans digest PDF

The problem is that this is still just retrieval and mechanical. In RAG, you split a PDF into small chunks, but this is way different from how humans digest PDFs. If I hire an RA to go through my Zotero lib and make a mind-map of sorts, he/she would combine papers, paragraphs, figures, etc. to come up with a "concepts" map, which is way richer than a retrieval system that merely finds the semantic similarity between my query and pieces of text. RAG is good for semantic search, but really we need something that works at a knowledge/understanding level as opposed to data/information level.

dmezzetti 3y ago
I think what you're looking for is possible with LLM agents. For paperai (mentioned previously) at least, it's possible to build workflows that connect multiple prompt steps together. txtai (included with paperai) has the ability to build semantic graphs (https://neuml.hashnode.dev/introducing-the-semantic-graph https://neuml.hashnode.dev/introducing-the-semantic-graph). I agree that RAG is just one part of the equation. But the tools are available if one wanted to build their own complex multi-agent workflow.
- alchemist1e9 3y ago
  In the example of the RA being automated by an LLM workflow by agents, I agree it’s very possible and it requires defining a set of specific agents, using prompts and allow function calling for tools, and then defining a full workflow between the agents. The workflow can likely be modeled by breaking down the individual steps the RA takes when doing their work. The agents are likely very narrow and specific, they do one very very specific task. Then the workflow is a DAG chaining their work together.
dartos 3y ago
There are lots of experiments around generating knowledge graph mutations and queries to build this kind of relational knowledge. In neo4j, for example, relations tend to have natural language names. (The cat BELONGS_TO the human) So LLMs appear to be apt at making those queries

dmezzetti 3y ago

I think what you're looking for is possible with LLM agents. For paperai (mentioned previously) at least, it's possible to build workflows that connect multiple prompt steps together. txtai (included with paperai) has the ability to build semantic graphs (https://neuml.hashnode.dev/introducing-the-semantic-graph https://neuml.hashnode.dev/introducing-the-semantic-graph). I agree that RAG is just one part of the equation. But the tools are available if one wanted to build their own complex multi-agent workflow.

alchemist1e9 3y ago
In the example of the RA being automated by an LLM workflow by agents, I agree it’s very possible and it requires defining a set of specific agents, using prompts and allow function calling for tools, and then defining a full workflow between the agents. The workflow can likely be modeled by breaking down the individual steps the RA takes when doing their work. The agents are likely very narrow and specific, they do one very very specific task. Then the workflow is a DAG chaining their work together.

dartos 3y ago

There are lots of experiments around generating knowledge graph mutations and queries to build this kind of relational knowledge. In neo4j, for example, relations tend to have natural language names. (The cat BELONGS_TO the human) So LLMs appear to be apt at making those queries