3 ms·

Is it possible Andreessen are misunderstanding how pinecone/vector dbs are used? It seems like they are pitching it as "memory for large language models" or som

Is it possible Andreessen are misunderstanding how pinecone/vector dbs are used? It seems like they are pitching it as "memory for large language models" or something. Are people using vector db's in some way I'm not aware of? To me it's a database to help you do a semantic search. A multi-token string is converted into a single embedding. Like maybe 1000 words into one embedding. This is helpful because you can quickly find the relevant parts of a document to answer a question and there are token limits into an LLM, but the idea that it's helping the LLM keep state or something seems off? Is it possible they are confusing the use of embeddings across whole swaths of text to do a semantic search with the embeddings that happen on a per token basis as data runs through an LLM? Same word, same basic idea, but used so differently that they may as well be different words?

darkteflon 4y ago
I might be mistaken, but my understanding from having played around with LangChain for a couple months is that because you’ve got to keep all your state in the context window, giving the model access to a vectorstore containing the entire chat history allows it to retrieve relevant messages against your query that can then be stuffed or mapreduced into the context window for the next response. The alternative - and I believe the way the ChatGPT web app currently works - is just to stuff/mapreduce user input and machine response into the context window on each step, which quickly gets quite lossy.
- kordlessagain 4y ago
  You aren't mistaken. Keeping state, or storing memories, is where it's at with prompts. The trick is knowing what to remember and what to forget. I consider vector engines to be "hot" models, given they are storing the vector representations of text already run through the "frozen" model. Having written something a while back that indexes documents and enters into discussion with them, I'm pretty sure ChatGPT is using some type of embedding lookup/match/distance on the history in the window. That means not all text is submitted at the next entry, but whatever mostly matches what is entered by the user (in vector space) is likely pulled in and sent over in the final prompt.
  - danielmarkbruce 4y ago
    Sure - but a vector db is helping you keep your prompts to under size X. It isn't adding state and there are various mechanisms to keep your prompt to under size X - like summarization, providing a table of contents etc. It seems to me that vector db and semantic search are one trick in a pile of tricks to keep prompt sizes down until we can get the input sizes up (although gpt4 already takes 32,000 tokens). Using semantic search to find relevant chunks seems misguided but practical in the short term. One of the key benefits of LLMs is they can take into account a lot of context.
    - darkteflon 4y ago
      Context constraint is a cheap way to keep the model on-topic. So rather than relying on an ever-growing context window to stuff/mapreduce more undifferentiated “context” (the entire chat history), interposing a vector search engine that only returns relevant context tends to get you better overall model performance, in addition to being scalable in a way that increasing context window size is not.
      - danielmarkbruce 4y ago
        Agreed. But summarization is better to keep the model on topic for most cases. And there are other tricks. Vectors and semantic search are one (likely questionable way given LLMs can likely reason over a table of contents or similar better) to search a large corpus or very large document. It's really only appropriate for a specific set of use cases. It's not some "general memory layer" for AI.
        appenz 4y ago
        Summarization is much more expensive than vector db's. Assume you have 1m tokens of context. You could run all through GPT-4 and summarize the information, but it would cost $60 (based on current prices) and take 10's of minutes of GPU time to do the inference. Disclaimer: I work for a16z and on the infra team, so consider me biassed.
- rahimnathwani 4y ago
  Yes, one of the memory options in Langchain is to use a vector store: https://python.langchain.com/en/latest/modules/memory/types/vectorstore_retriever_memory.html https://python.langchain.com/en/latest/modules/memory/types/... It also has a more basic version that just keeps a log of past messages. I don't know whether there's a way (or even a need) to combine these approaches. In a long conversation, it might be useful to trust more recent information more than earlier messages, but Langchain's vector memory doesn't care about sequence.

darkteflon 4y ago

I might be mistaken, but my understanding from having played around with LangChain for a couple months is that because you’ve got to keep all your state in the context window, giving the model access to a vectorstore containing the entire chat history allows it to retrieve relevant messages against your query that can then be stuffed or mapreduced into the context window for the next response. The alternative - and I believe the way the ChatGPT web app currently works - is just to stuff/mapreduce user input and machine response into the context window on each step, which quickly gets quite lossy.

kordlessagain 4y ago
You aren't mistaken. Keeping state, or storing memories, is where it's at with prompts. The trick is knowing what to remember and what to forget. I consider vector engines to be "hot" models, given they are storing the vector representations of text already run through the "frozen" model. Having written something a while back that indexes documents and enters into discussion with them, I'm pretty sure ChatGPT is using some type of embedding lookup/match/distance on the history in the window. That means not all text is submitted at the next entry, but whatever mostly matches what is entered by the user (in vector space) is likely pulled in and sent over in the final prompt.
- danielmarkbruce 4y ago
  Sure - but a vector db is helping you keep your prompts to under size X. It isn't adding state and there are various mechanisms to keep your prompt to under size X - like summarization, providing a table of contents etc. It seems to me that vector db and semantic search are one trick in a pile of tricks to keep prompt sizes down until we can get the input sizes up (although gpt4 already takes 32,000 tokens). Using semantic search to find relevant chunks seems misguided but practical in the short term. One of the key benefits of LLMs is they can take into account a lot of context.
  - darkteflon 4y ago
    Context constraint is a cheap way to keep the model on-topic. So rather than relying on an ever-growing context window to stuff/mapreduce more undifferentiated “context” (the entire chat history), interposing a vector search engine that only returns relevant context tends to get you better overall model performance, in addition to being scalable in a way that increasing context window size is not.
    - danielmarkbruce 4y ago
      Agreed. But summarization is better to keep the model on topic for most cases. And there are other tricks. Vectors and semantic search are one (likely questionable way given LLMs can likely reason over a table of contents or similar better) to search a large corpus or very large document. It's really only appropriate for a specific set of use cases. It's not some "general memory layer" for AI.
      - appenz 4y ago
        Summarization is much more expensive than vector db's. Assume you have 1m tokens of context. You could run all through GPT-4 and summarize the information, but it would cost $60 (based on current prices) and take 10's of minutes of GPU time to do the inference. Disclaimer: I work for a16z and on the infra team, so consider me biassed.
rahimnathwani 4y ago
Yes, one of the memory options in Langchain is to use a vector store: https://python.langchain.com/en/latest/modules/memory/types/vectorstore_retriever_memory.html https://python.langchain.com/en/latest/modules/memory/types/... It also has a more basic version that just keeps a log of past messages. I don't know whether there's a way (or even a need) to combine these approaches. In a long conversation, it might be useful to trust more recent information more than earlier messages, but Langchain's vector memory doesn't care about sequence.