7 ms·

Txtai: Open-source vector search and RAG for minimalists

249 points by dmezzetti 2y ago

dmezzetti 2y ago
Link to source (Apache 2.0): https://github.com/neuml/txtai https://github.com/neuml/txtai
freeqaz 2y ago
This looks interesting. I've been wanting to build some tools to help feed text documents into Stable Diffusion and this looks like it could be helpful. Are there any other libs people are aware of that they'd recommend in this space?
janice1999 2y ago
It's frustrating when developers of ML projects don't state even the most basic requirements. Do I need an Nvidia 4090 or a cluster of H100s to run this?
- dmezzetti 2y ago
  A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU. If you're using remote API services, you might be able to just use a CPU.
- malux85 2y ago
  It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency? How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?
  - tbrownaw 2y ago
    There's a difference between "how many CPU-hours will my task need" and "how much memory does this program use to even start up".
    - deleted 2y ago
      [deleted]
    - malux85 2y ago
      Having some idea of the task will guide the choice of model, which will be an enormous factor in memory use (I.e. whether it will startup or not) Do you need a 70b param model or a 7b model? Theres thousands and thousands of dollars hardware difference there With no idea of the task, one can’t even ball park it
      - tbrownaw 2y ago
        This particular tool has a page listing recommended models: https://neuml.github.io/txtai/models/ https://neuml.github.io/txtai/models/
- Der_Einzige 2y ago
  The embedding models at the heart of txtai can be small enough to run on intel CPUs from ten years ago. It's extremely frustrating when HN commentators don't do even the most basic research into the product that they are critiquing.
pjot 2y ago
I’ve done something similar, but using duckDB as the backend/vector store. You can use embeddings from wherever. My demo uses OpenAI. https://github.com/patricktrainer/duckdb-embedding-search https://github.com/patricktrainer/duckdb-embedding-search
dmezzetti 2y ago
Hello, author of txtai here. txtai was created back in 2020 starting with semantic search of medical literature. It has since grown into a framework for vector search, retrieval augmented generation (RAG) and large language model (LLM) orchestration/workflows. The goal of txtai is to be simple, performant, innovative and easy-to-use. It had vector search before many current projects existed. Semantic Graphs were added in 2022 before the Generative AI wave of 2023/2024. GraphRAG is a hot topic but txtai had examples of using graphs to build search contexts back in 2022/2023. There is a commitment to quality and performance, especially with local models. For example, it's vector embeddings component streams vectors to disk during indexing and uses mmaped arrays to enable indexing large datasets locally on a single node. txtai's BM25 component is built from the scratch to work efficiently in Python leading to 6x better memory utilization and faster search performance than the BM25 Python library most commonly used. I often see others complain about AI/LLM/RAG frameworks, so I wanted to share this project as many don't know it exists. Link to source (Apache 2.0): https://github.com/neuml/txtai https://github.com/neuml/txtai
fastneutron 2y ago
I’ve been building a RAG mini app with txtai these past few weeks and it’s been pretty smooth. I’m between this and llamaindex as the backend for a larger app I want to build for a small-to-midsize customer. With the (potentially) obvious bias towards your own framework, are there situations in which you would not recommend it for a particular application?
- dmezzetti 2y ago
  Glad to hear txtai is on your list. I recently wrote an article (https://medium.com/neuml/vector-search-rag-landscape-a-review-with-txtai-a7f37ad0e187 https://medium.com/neuml/vector-search-rag-landscape-a-revie...) comparing txtai with other popular frameworks. I was expecting to find some really interesting and innovative things in the others. But from my perspective I was underwhelmed. I'm a big fan of simplicity and none of them are following that strategy. Agentic workflows seem like a big fancy term but I don't see the value currently. Things are hard enough as it is. If your team is already using another framework, I'm sure anything can work. Some of the other projects are VC-backed with larger teams. In some cases, that may be important.
ipsi 2y ago
So here's something I've been wanting to do for a while, but have kinda been struggling to figure out _how_ to do it. txtai looks like it has all the tools necessary to do the job, I'm just not sure which tool(s), and how I'd use them. Basically, I'd like to be able to take PDFs of, say, D&D books, extract that data (this step is, at least, something I can already do), and load it into an LLM to be able to ask questions like: * What does the feat "Sentinel" do? * Who is Elminster? * Which God(s) do Elves worship in Faerûn? * Where I can I find the spell "Crusader's Mantle"? And so on. Given this data is all under copyright, I'd probably have to stick to using a local LLM to avoid problems. And, while I wouldn't expect it to have good answers to all (or possibly any!) of those questions, I'd nevertheless love to be able to give it a try. I'm just not sure where to start - I think I'd want to fine-tune an existing model since this is all natural language content, but I get a bit lost after that. Do I need to pre-process the content to add extra information that I can't fetch relatively automatically. e.g., page numbers are simple to add in, but would I need to mark out things like chapter/section headings, or in-character vs out-of-character text? Do I need to add all the content in as a series of questions and answers, like "What information is on page 52 of the Player's Handbook? => <text of page>"?
- dmezzetti 2y ago
  Based on what you're looking to do, it sounds like Retrieval Augmented Generation (RAG) should help. This article has an example on how to do that with txtai: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://neuml.hashnode.dev/build-rag-pipelines-with-txtai RAG sounds sophisticated but it's actually quite simple. For each question, a database (vector database, keyword, relational etc) is first searched. The top n results are then inserted into a prompt and that is what is run with the LLM. Before fine-tuning, I'd try that out first. I'm planning to have another example notebook out soon building on this.
  - ipsi 2y ago
    Ah, that's very helpful, thanks! I'll have a dig into this at some point relatively soon. An example of how I might provide references with page numbers or chapter names would be great (even if this means a more complex text-extraction pipeline). As would examples showing anything I can do to indicate differences that are obvious to me but that an LLM would be unlikely to pick up, such as the previously mentioned in-character vs out-of-character distinction. This is mostly relevant for asking questions about the setting, where in-character information might be suspect ("unreliable narrator"), while out-of-character information is generally fully accurate. Tangentially, is this something that I could reasonably experiment with without a GPU? While I do have a 4090, it's in my Windows gaming machine, which isn't really set up for AI/LLM/etc development.
anotherpaulg 2y ago
I did some prototyping with txtai for the RAG used in aider’s interactive help feature [0]. This lets users ask aider questions about using aider, customizing settings, troubleshooting, using LLMs, etc. I really liked the simplicity of txtai. But it seems to require Java as a dependency! Aider is an end user cli tool, and ultimately I couldn’t take on the support burden of asking my users to install Java. [0] https://aider.chat/docs/troubleshooting/support.html https://aider.chat/docs/troubleshooting/support.html
- dmezzetti 2y ago
  Thanks for giving txtai a try. txtai doesn't require Java. It has a text extraction component which can optionally use Apache Tika. Apache Tika is a Java library. Tika can also be spun up as a Docker image much like someone can spin up Ollama for LLM inference. Looking at your use case, it appears you wanted to parse and index HTML? If so, the only dependency should have been BeautifulSoup4. Alternatively, one can use another library such as unstructured.io or PyMuPDF for word/pdf. Those are not issue free though. For example, unstructured requires libreoffice for word documents, poppler for pdfs. PyMuPDF is AGPL, which is a non-starter for many. Apache Tika is Apache 2.0, mature and it has robust production-quality support for a lot of formats.
  - anotherpaulg 2y ago
    Thanks for the reply. I really did like the txtai approach. I am working with markdown files. I think that required me to use Tika & Java based on this note in your docs [0]? Note: BeautifulSoup4 only supports HTML documents, anything else requires Tika and Java to be installed. Tika did a great job of chunking the markdown into sections with appropriate parent header context, if I remember correctly. I just couldn't ask my users to manually install such complex dependencies. I worried about the support burden I would incur, due to the types of issues they would encounter. [0] https://neuml.github.io/txtai/pipeline/data/textractor/ https://neuml.github.io/txtai/pipeline/data/textractor/
    - dmezzetti 2y ago
staticautomatic 2y ago
Looks pretty cool! Is this intended to be a simple alternative to, say, cobbling together something with LangChain and Chroma?
- dmezzetti 2y ago
  Thanks. That is correct. This is an alternative to LangChain/LlamaIndex on the RAG side and Chroma on the vector db side.
sampling 2y ago
Has anyone had experience with qdrant (https://qdrant.tech/ https://qdrant.tech/) as a vector store data and can speak to how txtai compares?
- kennethwolters 2y ago
  txtai is not (just) a vector store, it's a full-fledged RAG system. Apples and oranges if you ask me.
  - dmezzetti 2y ago
    I agree that the comparison between langchain/llamaindex is probably the better one. With that being said, txtai has a much more in-depth approach with how it builds it's data stores vs just assuming the underlying systems will handle everything. It supports running SQL statements and integrates the components in a way other RAG systems don't. It was also a vector store before it had a RAG workflow. There are years of code behind that part.
    - kennethwolters 2y ago
      It is very impressive :)
v3ss0n 2y ago
Txtai get things done quick, but one problem is the code base is not properly typed (in contrast to a bit higher learning curve but more proper Haystack). Would be nice if this project is properly type annotated.
- dmezzetti 2y ago
  We certainly could add typing to the main API calls. Typing isn't a huge thing to me as a developer, so I've never really made it a priority. The only place there is typing is in the FastAPI hooks given it's required.
haolez 2y ago
"Interested in an easy and secure way to run hosted txtai applications? Then join the txtai.cloud preview to learn more." I wish the author all the best and this seems to be a very sane and minimalist approach when compared to all the other enterprise-backed frameworks and libraries in this space. I might even become a customer! However, has someone started an open source library that's fully driven by a community? I'm thinking of something like Airflow or Git. I'm not saying that the "purist" model is the best or enterprise-backed frameworks are evil. I'm just not seeing this type of project in this space.
- dmezzetti 2y ago
  Appreciate the well wishes. NeuML is not venture backed, so there is no impetus to build a hosted version. The main goal is making it easier for a larger audience.
antman 2y ago
What type of embeddings db does it use? Is it interchangeable?
- dmezzetti 2y ago
  You can read more on that here: https://neuml.github.io/txtai/embeddings/configuration/vectors/ https://neuml.github.io/txtai/embeddings/configuration/vecto... txtai supports Hugging Face Transformers models, llama.cpp embeddings models and API services such as OpenAI/Cohere/Ollama.

249 points by dmezzetti 2y ago

Link to source (Apache 2.0): https://github.com/neuml/txtai https://github.com/neuml/txtai

This looks interesting. I've been wanting to build some tools to help feed text documents into Stable Diffusion and this looks like it could be helpful. Are there any other libs people are aware of that they'd recommend in this space?

janice1999 2y ago

It's frustrating when developers of ML projects don't state even the most basic requirements. Do I need an Nvidia 4090 or a cluster of H100s to run this?

dmezzetti 2y ago
A RTX 3090 is more than enough for 7B LLMs. With 4-bit quantization, you can run inference with an even larger LLM using a 24GB GPU. If you're using remote API services, you might be able to just use a CPU.
malux85 2y ago
It’s frustrating when people ask for hardware requirements without stating what they are trying to do, do you have 100,000,000 books to index or do you have 5 articles? What are the context lengths you need? What about latency? How can someone tell you what hardware you need when you give literally no information about what you’re trying to do?
- tbrownaw 2y ago
  There's a difference between "how many CPU-hours will my task need" and "how much memory does this program use to even start up".
  - deleted 2y ago
    [deleted]
  - malux85 2y ago
    Having some idea of the task will guide the choice of model, which will be an enormous factor in memory use (I.e. whether it will startup or not) Do you need a 70b param model or a 7b model? Theres thousands and thousands of dollars hardware difference there With no idea of the task, one can’t even ball park it
    - tbrownaw 2y ago
      This particular tool has a page listing recommended models: https://neuml.github.io/txtai/models/ https://neuml.github.io/txtai/models/
Der_Einzige 2y ago
The embedding models at the heart of txtai can be small enough to run on intel CPUs from ten years ago. It's extremely frustrating when HN commentators don't do even the most basic research into the product that they are critiquing.

pjot 2y ago

I’ve done something similar, but using duckDB as the backend/vector store. You can use embeddings from wherever. My demo uses OpenAI. https://github.com/patricktrainer/duckdb-embedding-search https://github.com/patricktrainer/duckdb-embedding-search

dmezzetti 2y ago

Hello, author of txtai here. txtai was created back in 2020 starting with semantic search of medical literature. It has since grown into a framework for vector search, retrieval augmented generation (RAG) and large language model (LLM) orchestration/workflows. The goal of txtai is to be simple, performant, innovative and easy-to-use. It had vector search before many current projects existed. Semantic Graphs were added in 2022 before the Generative AI wave of 2023/2024. GraphRAG is a hot topic but txtai had examples of using graphs to build search contexts back in 2022/2023. There is a commitment to quality and performance, especially with local models. For example, it's vector embeddings component streams vectors to disk during indexing and uses mmaped arrays to enable indexing large datasets locally on a single node. txtai's BM25 component is built from the scratch to work efficiently in Python leading to 6x better memory utilization and faster search performance than the BM25 Python library most commonly used. I often see others complain about AI/LLM/RAG frameworks, so I wanted to share this project as many don't know it exists. Link to source (Apache 2.0): https://github.com/neuml/txtai https://github.com/neuml/txtai

fastneutron 2y ago

I’ve been building a RAG mini app with txtai these past few weeks and it’s been pretty smooth. I’m between this and llamaindex as the backend for a larger app I want to build for a small-to-midsize customer. With the (potentially) obvious bias towards your own framework, are there situations in which you would not recommend it for a particular application?

dmezzetti 2y ago
Glad to hear txtai is on your list. I recently wrote an article (https://medium.com/neuml/vector-search-rag-landscape-a-review-with-txtai-a7f37ad0e187 https://medium.com/neuml/vector-search-rag-landscape-a-revie...) comparing txtai with other popular frameworks. I was expecting to find some really interesting and innovative things in the others. But from my perspective I was underwhelmed. I'm a big fan of simplicity and none of them are following that strategy. Agentic workflows seem like a big fancy term but I don't see the value currently. Things are hard enough as it is. If your team is already using another framework, I'm sure anything can work. Some of the other projects are VC-backed with larger teams. In some cases, that may be important.

ipsi 2y ago

So here's something I've been wanting to do for a while, but have kinda been struggling to figure out _how_ to do it. txtai looks like it has all the tools necessary to do the job, I'm just not sure which tool(s), and how I'd use them. Basically, I'd like to be able to take PDFs of, say, D&D books, extract that data (this step is, at least, something I can already do), and load it into an LLM to be able to ask questions like: * What does the feat "Sentinel" do? * Who is Elminster? * Which God(s) do Elves worship in Faerûn? * Where I can I find the spell "Crusader's Mantle"? And so on. Given this data is all under copyright, I'd probably have to stick to using a local LLM to avoid problems. And, while I wouldn't expect it to have good answers to all (or possibly any!) of those questions, I'd nevertheless love to be able to give it a try. I'm just not sure where to start - I think I'd want to fine-tune an existing model since this is all natural language content, but I get a bit lost after that. Do I need to pre-process the content to add extra information that I can't fetch relatively automatically. e.g., page numbers are simple to add in, but would I need to mark out things like chapter/section headings, or in-character vs out-of-character text? Do I need to add all the content in as a series of questions and answers, like "What information is on page 52 of the Player's Handbook? => <text of page>"?

dmezzetti 2y ago
Based on what you're looking to do, it sounds like Retrieval Augmented Generation (RAG) should help. This article has an example on how to do that with txtai: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai https://neuml.hashnode.dev/build-rag-pipelines-with-txtai RAG sounds sophisticated but it's actually quite simple. For each question, a database (vector database, keyword, relational etc) is first searched. The top n results are then inserted into a prompt and that is what is run with the LLM. Before fine-tuning, I'd try that out first. I'm planning to have another example notebook out soon building on this.
- ipsi 2y ago
  Ah, that's very helpful, thanks! I'll have a dig into this at some point relatively soon. An example of how I might provide references with page numbers or chapter names would be great (even if this means a more complex text-extraction pipeline). As would examples showing anything I can do to indicate differences that are obvious to me but that an LLM would be unlikely to pick up, such as the previously mentioned in-character vs out-of-character distinction. This is mostly relevant for asking questions about the setting, where in-character information might be suspect ("unreliable narrator"), while out-of-character information is generally fully accurate. Tangentially, is this something that I could reasonably experiment with without a GPU? While I do have a 4090, it's in my Windows gaming machine, which isn't really set up for AI/LLM/etc development.

anotherpaulg 2y ago

I did some prototyping with txtai for the RAG used in aider’s interactive help feature [0]. This lets users ask aider questions about using aider, customizing settings, troubleshooting, using LLMs, etc. I really liked the simplicity of txtai. But it seems to require Java as a dependency! Aider is an end user cli tool, and ultimately I couldn’t take on the support burden of asking my users to install Java. [0] https://aider.chat/docs/troubleshooting/support.html https://aider.chat/docs/troubleshooting/support.html

dmezzetti 2y ago
Thanks for giving txtai a try. txtai doesn't require Java. It has a text extraction component which can optionally use Apache Tika. Apache Tika is a Java library. Tika can also be spun up as a Docker image much like someone can spin up Ollama for LLM inference. Looking at your use case, it appears you wanted to parse and index HTML? If so, the only dependency should have been BeautifulSoup4. Alternatively, one can use another library such as unstructured.io or PyMuPDF for word/pdf. Those are not issue free though. For example, unstructured requires libreoffice for word documents, poppler for pdfs. PyMuPDF is AGPL, which is a non-starter for many. Apache Tika is Apache 2.0, mature and it has robust production-quality support for a lot of formats.
- anotherpaulg 2y ago
  Thanks for the reply. I really did like the txtai approach. I am working with markdown files. I think that required me to use Tika & Java based on this note in your docs [0]? Note: BeautifulSoup4 only supports HTML documents, anything else requires Tika and Java to be installed. Tika did a great job of chunking the markdown into sections with appropriate parent header context, if I remember correctly. I just couldn't ask my users to manually install such complex dependencies. I worried about the support burden I would incur, due to the types of issues they would encounter. [0] https://neuml.github.io/txtai/pipeline/data/textractor/ https://neuml.github.io/txtai/pipeline/data/textractor/
  - dmezzetti 2y ago

staticautomatic 2y ago

Looks pretty cool! Is this intended to be a simple alternative to, say, cobbling together something with LangChain and Chroma?

dmezzetti 2y ago
Thanks. That is correct. This is an alternative to LangChain/LlamaIndex on the RAG side and Chroma on the vector db side.

sampling 2y ago

Has anyone had experience with qdrant (https://qdrant.tech/ https://qdrant.tech/) as a vector store data and can speak to how txtai compares?

kennethwolters 2y ago
txtai is not (just) a vector store, it's a full-fledged RAG system. Apples and oranges if you ask me.
- dmezzetti 2y ago
  I agree that the comparison between langchain/llamaindex is probably the better one. With that being said, txtai has a much more in-depth approach with how it builds it's data stores vs just assuming the underlying systems will handle everything. It supports running SQL statements and integrates the components in a way other RAG systems don't. It was also a vector store before it had a RAG workflow. There are years of code behind that part.
  - kennethwolters 2y ago
    It is very impressive :)

v3ss0n 2y ago

Txtai get things done quick, but one problem is the code base is not properly typed (in contrast to a bit higher learning curve but more proper Haystack). Would be nice if this project is properly type annotated.

dmezzetti 2y ago
We certainly could add typing to the main API calls. Typing isn't a huge thing to me as a developer, so I've never really made it a priority. The only place there is typing is in the FastAPI hooks given it's required.

haolez 2y ago

"Interested in an easy and secure way to run hosted txtai applications? Then join the txtai.cloud preview to learn more." I wish the author all the best and this seems to be a very sane and minimalist approach when compared to all the other enterprise-backed frameworks and libraries in this space. I might even become a customer! However, has someone started an open source library that's fully driven by a community? I'm thinking of something like Airflow or Git. I'm not saying that the "purist" model is the best or enterprise-backed frameworks are evil. I'm just not seeing this type of project in this space.

dmezzetti 2y ago
Appreciate the well wishes. NeuML is not venture backed, so there is no impetus to build a hosted version. The main goal is making it easier for a larger audience.

antman 2y ago

What type of embeddings db does it use? Is it interchangeable?

dmezzetti 2y ago
You can read more on that here: https://neuml.github.io/txtai/embeddings/configuration/vectors/ https://neuml.github.io/txtai/embeddings/configuration/vecto... txtai supports Hugging Face Transformers models, llama.cpp embeddings models and API services such as OpenAI/Cohere/Ollama.