8 ms·

Memary: Open-Source Longterm Memory for Autonomous Agents

216 points by james_chu 2y ago

mlvljr 2y ago
[dead]
CuriouslyC 2y ago
While I'm 100% on board with RAG using associative memory, I'm not sure you need Neo4J. Associative recall is generally going to be one level deep, and you're doing a top K cut so even if it wasn't the second order associations are probably not going to make the relevance cut. This could be done relationally, and then if you're using pg_vector you could retrieve all your rag contents in one query.
- verdverm 2y ago
  My initial thought was "building the knowledge graph is what LLMs and the embedding process does implicitly", why the need for a graphdb like Neo4j?
  - throwaway11460 2y ago
    So your solution would be to fine tune the LLM with new knowledge? How do you make sure it preserves all facts and connections/relations and how can you verify during runtime it actually did, and didn't introduce false memories/connections in the process?
    - brigadier132 2y ago
      I think you are misunderstanding. An embedding places a piece of knowledge in N dimensional space. By using vector distance search you are already getting conceptually similar results.
      - verdverm 2y ago
        Not only that, the embedding process can represent relationships and ontology, especially the subtle aspects that are hard to capture in edges or json Go back to the original word2vec example, how would you put this in neo4j, in generalization? king - man + woman = queen
        dsabanin 2y ago
        I like your answer and a great example of the limitation of knowledge/semantic graphs. Personally, I'd put a knowledge graph on top of the responses to expose it to LLM as an authority & frame of reference. I think it should be an effective form of protection against hallucination and preventing outright incorrect / harmful outputs in contradiction with the facts known by the graph. At least in my experiments.
        threecheese 2y ago
        Is it not valuable though to differentiate statistically inferred relationships from those that are user-declared? I would think that a) these are complementary, and b) the potential inaccuracy of the former is much larger than the latter.
        verdverm 2y ago
        Absolutely, however this project says > Llamaindex was used to add nodes into the graph store based on documents. So it sounds like they are generating it based on LLM output rather then user-defined. I also wonder how often you need more than a single hop that graphdbs aim to speed up. In an agent system with self-checking and reranking, you're going to be performing multiple queries anyhow There is also interesting research around embedding graphs that overlaps with these ideas.
        esafak 2y ago
        Topological relationships vs metric relationships. I suppose a great embedding could handle both, but a graph database might help in the tail, where the quality of the embeddings is weaker?
    - verdverm 2y ago
      I think RAG has a lot to say here. New content / facts go through the embedding process and are then available for query. I don't generally disagree that a more discrete (not continuous) knowledge base will be another component to augment ai systems. The harder part is how do you build this? (Curate, clean, ETL, query) Not sure a graphdb is the best first choice. Relational DBs can take you pretty far and it is unclear how many 1+N or multi-hop queries you'll need in a robust ai / agent system
  - dbish 2y ago
    I dont know if you need a graphdb in particular but there are likely explicit relationships or entities to resolve to eachother that you’d want to add that aren’t known by a general model about your use case. For example if you are personalizing an assistant maybe you need to represent that “John” in the contacts app is the same as “Jdubs” in Instagram and is this person’s husband.
- snorkel 2y ago
  LLMs have a limited context size, i.e. the chat bot can only recall so much of the conversation. This project is building a knowledge graph of the entire conversation(s), then using that knowledge graph as a RAG database.
  - kingJulio 2y ago
    Exactly! With memary only relevant information is passed into the finite context window.
- gaogao 2y ago
  I think there's a lot of cases where you don't want to just RAG it. If you're going for tool assisted, it's pretty neat to have agent write out queries for what it needs against the knowledge graph. There was an article recently about how LLMs are bad at inferring B is A from A is B. You can also do more precise math against it, which is useful for questions even people need to reason out. I need to dig into what they're doing here more with their approach, but I think using an LLM for both producing and consuming a knowledge graph is pretty nifty, which I wrote up about a year ago here, https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... . I will say figuring out how to actually add that conversation properly into a large knowledge graph is a bit tricky. ML does seem slightly better at producing an ontology than humans though (look how many times we've had to revise scientific names for creatures or book ordering)
  - dbish 2y ago
    Yes, but this doesn’t seem to be an actual knowledge graph which is part of the issue imho. If you look at the Microsoft knowledge graph paper linked in the repo it looks like they build out a real entity-relationship based knowledge graph rather then storing responses and surface form text directly.
  - jjfoooo6 2y ago
    I think it's relatively unlikely that having an agent write graph queries will outperform vector search against graph information outputted into text and then transformed into vectors. The related issue that I think is being conflated in this thread is that even if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db
    - gaogao 2y ago
      Outperform in what way? There's some distinct things it already does better on like multi-hop and aggregate reasoning than a similarity context window dump. In general, tool-assisted, of which KG querying is one tool, does pretty good on the benchmarks and many of the LLM chats cutting over to it as the default. > if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db Postgres and MySQL do have pretty reasonable graph query extensions/features. If by easier, you mean effort to get up a MVP, I'd agree, but I'm a bit more dubious on the scale up, as you'd probably get something like Facebook and Tao.
abrichr 2y ago
Very interesting, thank you for making this available! At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt) we are looking into using pm4py (https://github.com/pm4py https://github.com/pm4py) to extract a process graph from a recording of user actions. I will look into this more closely. In the meantime, could the authors share their perspective on whether Memary could be useful here?
- oulipo 2y ago
  Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/ https://withaqua.com/ Perhaps it could make sense to add this to your effort?
  - abrichr 2y ago
    Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346 https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood — can you please clarify your suggestion?
    - oulipo 2y ago
      It's a kind of text input which mixes text and edition instructions, look at the demo
- dbish 2y ago
  What’s the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but haven’t found value in adding structure beyond representing them semantically in a list with the context around them of what happened.
  - abrichr 2y ago
    The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task. There's a lot of literature around process mining, e.g.: - https://en.wikipedia.org/wiki/Process_mining https://en.wikipedia.org/wiki/Process_mining - https://www.sciencedirect.com/science/article/pii/S2665963823000933 https://www.sciencedirect.com/science/article/pii/S266596382... - https://arxiv.org/abs/2404.06035 https://arxiv.org/abs/2404.06035
falcor84 2y ago
This is a really cool project, but is it just me that feels slightly uncomfortable with its name sounding so similar to "mammary"?
- Mtinie 2y ago
  Discomfort noted, but I think it can work in either case. Pronounced your way, it’s the proverbial teat of knowledge for LLMs.
  - jprete 2y ago
    Argh, that only makes it worse.
- wholinator2 2y ago
  Yeah, they could've gone with "Memury" which is pronounced much closer (at least for me) to the original "Memory".
  - kingJulio 2y ago
    The a is for agents :)
altilunium 2y ago
Sounds promising. Can this system be integrated with the Wikidata knowledge graph instead?
- kingJulio 2y ago
  Yes! You can easily swap knowledge graphs under the same agent. Would love to see this happen!
CyberDildonics 2y ago
How many times are people going to reinvent, rename and resell a database?
TrueDuality 2y ago
This seems like its overloading the term knowledge graph from its origins. Rather than having information and facts encoded into the graph, this appears to be a sort of similarity search over complete responses. It's blog style "related content" links to documents rather than encoded facts. Searching through their sources, it looks like the problem came from Neo4j's blog post misclassifying "knowledge augmentation" from a Microsoft research paper with "knowledge graph" (because of course they had to add "graph" to the title). This approach is fine, and probably useful but its not a knowledge graph in the sense that its structure isn't encoding anything about why or how different entities are actually related. A concrete example in a knowledge graph you might have an entity "Joe" and a separate entity "Paris". Joe is currently located in Paris so would have a typed edge between the two entities of something like "LocatedAt". I didn't dive into the code but what I inferred from the description and referenced literature, it is instead storing complete responses as "entities" and simply doing RAG style similarity searches to other nodes. It's a graph structured search index for sure but not a knowledge graph by the standard definitions.
- dbish 2y ago
  Exactly. Glad to see this. I do think knowledge graphs are important to AI assistants and agents though and someone needs to build a knowledge graph solution for that space. The idea of actual entities and relationships defined like triples with some schema and appropriately resolved and linked can be useful for querying and building up the right context. It may even be time to start bringing back some ideas from the schema.org back the day to standardize across agents/assistants what entities and actions are represented in data fed to them.
  - gaogao 2y ago
    Yeah, one of the specific things I'd love to do is collaboratively bulking up WikiData more. It's missing a ton of low hanging fruit that people using an ML augmented tool could really make some good progress on, similar to ML assisted OpenStreetMapping work
    - 2y ago
392 2y ago
Log4j was so unbelievably slow to load data, bloated, and hard to get working on my corporate managed box that I wasn't too sad when it turned out unable to handle my workload. Then the security team asked me why it was phoning home every 30 seconds. Ugh. I have since found Kuzu Db, which looks foundationally miles ahead. Plus no jvm. But have not yet given it a shot for rough edges. At the time, it was easier just to stay in plain application code. Hopefully the workload intended by this tool won't notice the bloat. But it would be nice to be able to dump huge loads of data into this knowledge graph as well, and let the GPT generate queries against it.
ec109685 2y ago
These new systems would do well to have a compelling “wow, this solves a hard problem that can’t be solved in another straightforward way”. The current YouTube video has a query about the Dallas Mavericks and it’s not clear how it’s using any of its memory or special machinery to answer the query: https://www.youtube.com/watch?v=GnUU3_xK6bg https://www.youtube.com/watch?v=GnUU3_xK6bg
- kingJulio 2y ago
  If you search about the Mavericks again (not included in the video) the agent will query the knowledge graph for results from prior executions.
anoy8888 2y ago
How does it compare with zep ai ? Anyone knows ?
- kingJulio 2y ago
  It's open source :)
BirbSingularity 2y ago
I hate when I find a cool AI project and I open the github to read the setup instructions and see "insert OpenAI API key." Nothing will make me loose interest faster.
- throwup238 2y ago
  Most projects also give you the option of providing an base url for the API so that people can use Azure's endpoints. You can use that config option with LiteLLM or a similar proxy tool to provide an OpenAI compatible interface for other models, whether that's a competitor like Claude or a local model like Llama or Mistral.
- muratsu 2y ago
  Is the expectation for the lib (or project) to work with various vendors or you expect to just pay for tokens
- kingJulio 2y ago
  You can easily incorporate llama 3 or other OS models
- api_or_ipa 2y ago
  Unconstructive comment. OpenAI is the golden standard for an llm: if you cared to dig deeper you’d realize that that you really could incorporate another llm with little effort.
JabavuAdams 2y ago
Looks cool. This is similar to what I'm doing for long-term memory in AISH, but packaged up nicely. Others have pointed out that they're somewhat abusing the term KG. But ... you could imagine other processes poring over the "raw" text chunks and building up a true KG from that.

216 points by james_chu 2y ago

[dead]

While I'm 100% on board with RAG using associative memory, I'm not sure you need Neo4J. Associative recall is generally going to be one level deep, and you're doing a top K cut so even if it wasn't the second order associations are probably not going to make the relevance cut. This could be done relationally, and then if you're using pg_vector you could retrieve all your rag contents in one query.

verdverm 2y ago
My initial thought was "building the knowledge graph is what LLMs and the embedding process does implicitly", why the need for a graphdb like Neo4j?
- throwaway11460 2y ago
  So your solution would be to fine tune the LLM with new knowledge? How do you make sure it preserves all facts and connections/relations and how can you verify during runtime it actually did, and didn't introduce false memories/connections in the process?
  - brigadier132 2y ago
    I think you are misunderstanding. An embedding places a piece of knowledge in N dimensional space. By using vector distance search you are already getting conceptually similar results.
    - verdverm 2y ago
      Not only that, the embedding process can represent relationships and ontology, especially the subtle aspects that are hard to capture in edges or json Go back to the original word2vec example, how would you put this in neo4j, in generalization? king - man + woman = queen
      - dsabanin 2y ago
        I like your answer and a great example of the limitation of knowledge/semantic graphs. Personally, I'd put a knowledge graph on top of the responses to expose it to LLM as an authority & frame of reference. I think it should be an effective form of protection against hallucination and preventing outright incorrect / harmful outputs in contradiction with the facts known by the graph. At least in my experiments.
      - threecheese 2y ago
        Is it not valuable though to differentiate statistically inferred relationships from those that are user-declared? I would think that a) these are complementary, and b) the potential inaccuracy of the former is much larger than the latter.
        verdverm 2y ago
        Absolutely, however this project says > Llamaindex was used to add nodes into the graph store based on documents. So it sounds like they are generating it based on LLM output rather then user-defined. I also wonder how often you need more than a single hop that graphdbs aim to speed up. In an agent system with self-checking and reranking, you're going to be performing multiple queries anyhow There is also interesting research around embedding graphs that overlaps with these ideas.
      - esafak 2y ago
        Topological relationships vs metric relationships. I suppose a great embedding could handle both, but a graph database might help in the tail, where the quality of the embeddings is weaker?
  - verdverm 2y ago
    I think RAG has a lot to say here. New content / facts go through the embedding process and are then available for query. I don't generally disagree that a more discrete (not continuous) knowledge base will be another component to augment ai systems. The harder part is how do you build this? (Curate, clean, ETL, query) Not sure a graphdb is the best first choice. Relational DBs can take you pretty far and it is unclear how many 1+N or multi-hop queries you'll need in a robust ai / agent system
- dbish 2y ago
  I dont know if you need a graphdb in particular but there are likely explicit relationships or entities to resolve to eachother that you’d want to add that aren’t known by a general model about your use case. For example if you are personalizing an assistant maybe you need to represent that “John” in the contacts app is the same as “Jdubs” in Instagram and is this person’s husband.
snorkel 2y ago
LLMs have a limited context size, i.e. the chat bot can only recall so much of the conversation. This project is building a knowledge graph of the entire conversation(s), then using that knowledge graph as a RAG database.
- kingJulio 2y ago
  Exactly! With memary only relevant information is passed into the finite context window.
gaogao 2y ago
I think there's a lot of cases where you don't want to just RAG it. If you're going for tool assisted, it's pretty neat to have agent write out queries for what it needs against the knowledge graph. There was an article recently about how LLMs are bad at inferring B is A from A is B. You can also do more precise math against it, which is useful for questions even people need to reason out. I need to dig into what they're doing here more with their approach, but I think using an LLM for both producing and consuming a knowledge graph is pretty nifty, which I wrote up about a year ago here, https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... . I will say figuring out how to actually add that conversation properly into a large knowledge graph is a bit tricky. ML does seem slightly better at producing an ontology than humans though (look how many times we've had to revise scientific names for creatures or book ordering)
- dbish 2y ago
  Yes, but this doesn’t seem to be an actual knowledge graph which is part of the issue imho. If you look at the Microsoft knowledge graph paper linked in the repo it looks like they build out a real entity-relationship based knowledge graph rather then storing responses and surface form text directly.
- jjfoooo6 2y ago
  I think it's relatively unlikely that having an agent write graph queries will outperform vector search against graph information outputted into text and then transformed into vectors. The related issue that I think is being conflated in this thread is that even if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db
  - gaogao 2y ago
    Outperform in what way? There's some distinct things it already does better on like multi-hop and aggregate reasoning than a similarity context window dump. In general, tool-assisted, of which KG querying is one tool, does pretty good on the benchmarks and many of the LLM chats cutting over to it as the default. > if your goal was to directly support graph queries, you could accomplish this with a vanilla database much easier than running a specialized graph db Postgres and MySQL do have pretty reasonable graph query extensions/features. If by easier, you mean effort to get up a MVP, I'd agree, but I'm a bit more dubious on the scale up, as you'd probably get something like Facebook and Tao.

abrichr 2y ago

Very interesting, thank you for making this available! At OpenAdapt (https://github.com/OpenAdaptAI/OpenAdapt https://github.com/OpenAdaptAI/OpenAdapt) we are looking into using pm4py (https://github.com/pm4py https://github.com/pm4py) to extract a process graph from a recording of user actions. I will look into this more closely. In the meantime, could the authors share their perspective on whether Memary could be useful here?

oulipo 2y ago
Very cool project! I think one of the main way (a bit orthogonal to what you do now) to adapt to GUI / CLI would be to develop an open-source version of something like Aqua Voice https://withaqua.com/ https://withaqua.com/ Perhaps it could make sense to add this to your effort?
- abrichr 2y ago
  Thanks! OpenAdapt already supports audio recording during demonstration (https://github.com/OpenAdaptAI/OpenAdapt/pull/346 https://github.com/OpenAdaptAI/OpenAdapt/pull/346). Perhaps I misunderstood — can you please clarify your suggestion?
  - oulipo 2y ago
    It's a kind of text input which mixes text and edition instructions, look at the demo
dbish 2y ago
What’s the goal of creating a graph from the actions? Do you have any related papers that talk about that? We also capture and learn from actions but haven’t found value in adding structure beyond representing them semantically in a list with the context around them of what happened.
- abrichr 2y ago
  The goal is to have a deterministic representation of a process that can be traversed in order to accomplish a task. There's a lot of literature around process mining, e.g.: - https://en.wikipedia.org/wiki/Process_mining https://en.wikipedia.org/wiki/Process_mining - https://www.sciencedirect.com/science/article/pii/S2665963823000933 https://www.sciencedirect.com/science/article/pii/S266596382... - https://arxiv.org/abs/2404.06035 https://arxiv.org/abs/2404.06035

falcor84 2y ago

This is a really cool project, but is it just me that feels slightly uncomfortable with its name sounding so similar to "mammary"?

Mtinie 2y ago
Discomfort noted, but I think it can work in either case. Pronounced your way, it’s the proverbial teat of knowledge for LLMs.
- jprete 2y ago
  Argh, that only makes it worse.
wholinator2 2y ago
Yeah, they could've gone with "Memury" which is pronounced much closer (at least for me) to the original "Memory".
- kingJulio 2y ago
  The a is for agents :)

altilunium 2y ago

Sounds promising. Can this system be integrated with the Wikidata knowledge graph instead?

kingJulio 2y ago
Yes! You can easily swap knowledge graphs under the same agent. Would love to see this happen!

CyberDildonics 2y ago

How many times are people going to reinvent, rename and resell a database?

TrueDuality 2y ago

This seems like its overloading the term knowledge graph from its origins. Rather than having information and facts encoded into the graph, this appears to be a sort of similarity search over complete responses. It's blog style "related content" links to documents rather than encoded facts. Searching through their sources, it looks like the problem came from Neo4j's blog post misclassifying "knowledge augmentation" from a Microsoft research paper with "knowledge graph" (because of course they had to add "graph" to the title). This approach is fine, and probably useful but its not a knowledge graph in the sense that its structure isn't encoding anything about why or how different entities are actually related. A concrete example in a knowledge graph you might have an entity "Joe" and a separate entity "Paris". Joe is currently located in Paris so would have a typed edge between the two entities of something like "LocatedAt". I didn't dive into the code but what I inferred from the description and referenced literature, it is instead storing complete responses as "entities" and simply doing RAG style similarity searches to other nodes. It's a graph structured search index for sure but not a knowledge graph by the standard definitions.

dbish 2y ago
Exactly. Glad to see this. I do think knowledge graphs are important to AI assistants and agents though and someone needs to build a knowledge graph solution for that space. The idea of actual entities and relationships defined like triples with some schema and appropriately resolved and linked can be useful for querying and building up the right context. It may even be time to start bringing back some ideas from the schema.org back the day to standardize across agents/assistants what entities and actions are represented in data fed to them.
- gaogao 2y ago
  Yeah, one of the specific things I'd love to do is collaboratively bulking up WikiData more. It's missing a ton of low hanging fruit that people using an ML augmented tool could really make some good progress on, similar to ML assisted OpenStreetMapping work
  - 2y ago

392 2y ago

Log4j was so unbelievably slow to load data, bloated, and hard to get working on my corporate managed box that I wasn't too sad when it turned out unable to handle my workload. Then the security team asked me why it was phoning home every 30 seconds. Ugh. I have since found Kuzu Db, which looks foundationally miles ahead. Plus no jvm. But have not yet given it a shot for rough edges. At the time, it was easier just to stay in plain application code. Hopefully the workload intended by this tool won't notice the bloat. But it would be nice to be able to dump huge loads of data into this knowledge graph as well, and let the GPT generate queries against it.

ec109685 2y ago

These new systems would do well to have a compelling “wow, this solves a hard problem that can’t be solved in another straightforward way”. The current YouTube video has a query about the Dallas Mavericks and it’s not clear how it’s using any of its memory or special machinery to answer the query: https://www.youtube.com/watch?v=GnUU3_xK6bg https://www.youtube.com/watch?v=GnUU3_xK6bg

kingJulio 2y ago
If you search about the Mavericks again (not included in the video) the agent will query the knowledge graph for results from prior executions.

anoy8888 2y ago

How does it compare with zep ai ? Anyone knows ?

kingJulio 2y ago
It's open source :)

BirbSingularity 2y ago

I hate when I find a cool AI project and I open the github to read the setup instructions and see "insert OpenAI API key." Nothing will make me loose interest faster.

throwup238 2y ago
Most projects also give you the option of providing an base url for the API so that people can use Azure's endpoints. You can use that config option with LiteLLM or a similar proxy tool to provide an OpenAI compatible interface for other models, whether that's a competitor like Claude or a local model like Llama or Mistral.
muratsu 2y ago
Is the expectation for the lib (or project) to work with various vendors or you expect to just pay for tokens
kingJulio 2y ago
You can easily incorporate llama 3 or other OS models
api_or_ipa 2y ago
Unconstructive comment. OpenAI is the golden standard for an llm: if you cared to dig deeper you’d realize that that you really could incorporate another llm with little effort.

JabavuAdams 2y ago

Looks cool. This is similar to what I'm doing for long-term memory in AISH, but packaged up nicely. Others have pointed out that they're somewhat abusing the term KG. But ... you could imagine other processes poring over the "raw" text chunks and building up a true KG from that.