8 ms·
So you wanna build a local RAG?
- mips_avatar 11mo agoOne thing I didn’t see here that might be hurting your performance is a lack of semantic chunking. It sounds like you’re embedding entire docs, which kind of breaks down if the docs contain multiple concepts. A better approach for recall is using some kind of chunking program to get semantic chunks (I like spacy though you have to configure it a bit). Then once you have your chunks you need to append context to how this chunk relates to the rest of your doc before you do your embedding. I have found anthropics approach to contextual retrieval to be very performant in my RAG systems (https://www.anthropic.com/engineering/contextual-retrieval https://www.anthropic.com/engineering/contextual-retrieval) you can just use gpt oss 20b as the model for generation of context. Unless I’ve misunderstood your post and you are doing some form of this in your pipeline you should see a dramatic improvement in performance once you implement this.
- yakkomajuri 11mo agohey, author (not op) here. we do do semantic chunking! I think maybe I gave the impression that we don't because of the mention of aggregating context but I tested this with questions that would require aggregating context from 15+ documents (meaning 2x that in chunks), hence the comment in the post!
- mips_avatar 11mo agoAh so you’re generating context from multiple docs for your chunks? How do you decide which docs get aggregated?
- nostrebored 11mo agoHaven’t seen an answer better than “vibes” here. Especially with data across multiple domains.
- mips_avatar 11mo agoI mean as long as they're not too long I suppose you could use just about any heuristic for grouping sources. Just seems like it would be hard to generate succinct context if you mess it up.
- NebulaStorm456 11mo agoIs there a way to convert documents into a hierarchical connected graph data structure which references each other similar to how we use personal knowledge tools like Obsidian and ability to traverse this graph? Is GraphRag technique trying to do this exactly?
- mips_avatar 11mo agoNot exactly what you’re looking for but Wilson Lin’s search engine creates a graph from the DOM for context. Here’s his write up: https://blog.wilsonl.in/search-engine/ https://blog.wilsonl.in/search-engine/
- simonw 11mo agoMy advice for building something like this: don't get hung up on a need for vector databases and embedding. Full text search or even grep/rg are a lot faster and cheaper to work with - no need to maintain a vector database index - and turn out to work really well if you put them in some kind of agentic tool loop. The big benefit of semantic search was that it could handle fuzzy searching - returning results that mention dogs if someone searches for canines, for example. Give a good LLM a search tool and it can come up with searches like "dog OR canine" on its own - and refine those queries over multiple rounds of searches. Plus it means you don't have to solve the chunking problem!
- leetrout 11mo agoSimon have you ever given a talk or written about this sort of pragmatism? A spin on how to achieve this with Datasette is an easy thing to imagine IMO.
- simonw 11mo agoI did a livestream thing about building RAG against FTS search in Datasette last year: https://simonwillison.net/2024/Jun/21/search-based-rag/ https://simonwillison.net/2024/Jun/21/search-based-rag/
- tra3 11mo agoI built a simple emacs package based on this idea [0]. It works surprisingly well, but I dont know how far it scales. It's likely not as frugal from a token usage perspective. 0: https://github.com/dmitrym0/dm-gptel-simple-org-memory https://github.com/dmitrym0/dm-gptel-simple-org-memory
- enraged_camel 11mo agoYes, exactly. We have our AI feature configured to use our pre-existing TypeSense integration and it's stunningly competent at figuring out exactly what search queries to use across which collections in order to find relevant results.
- 11mo ago
- nilirl 11mo agoWhy is it implicit that semantic search will outperform lexical search? Back in 2023 when I compared semantic search to lexical search (tantivy; BM25), I found the search results to be marginally different. Even if semantic search has slightly more recall, does the problem of context warrant this multi-component, homebrew search engine approach? By what important measure does it outperform a lexical search engine? Is the engineering time worth it?
- andoando 11mo agoThe benefit I see is you can have queries like "conversations between two scientists". Its very dependent on use case imo
- mips_avatar 11mo agoDepends on how important keyword matching vs something more ambiguous is to your app. In Wanderfugl there’s a bunch of queries where semantic search can find an important chunk that lacks a high bm25 score. The good news is you can get all the benefits of bm25 and semantic with a hybrid ranking. The answer isn’t one or the other.
- kgeist 11mo agoIt depends on how you test it. I recently found that the way devs test it differs radically from how users actually use it. When we first built our RAG, it showed promising results (around 90% recall on large knowledge bases). However, when the first actual users tried it, it could barely answer anything (closer to 30%). It turned out we relied on exact keywords too much when testing it: we knew the test knowledge base, so we formulated our questions in a way that helped the RAG find what we expected it to find. Real users don't know the exact terminology used in the articles. We had to rethink the whole thing. Lexical search is certainly not enough. Sure, you can run an agent on top of it, but that blows up latency - users aren't happy when they have to wait more than a couple of seconds.
- babelfish 11mo agoHow did you end up changing it? Creating new evals to measure the actual user experience seems easy enough, how did that inform your stack?
- barbazoo 11mo ago> What that means is that when you're looking to build a fully local RAG setup, you'll need to substitute whatever SaaS providers you're using for a local option for each of those components. Even starting with having "just" the documents and vector db locally is a huge first step and much more doable than going with a local LLM at the same time. I don't know any one or any org that has the resources to run their own LLM at scale.
- procaryote 11mo agoAren't there a bunch of models that run OK on consumer hardware now?
- lukan 11mo agoHopefully my new GPU will arrive tomorrow, then I can confirm myself, but if you look around online, there are lots of private people out there running their own models. A 16 GB GPU starts at 270€, which lets you run something like deepseek r.14, 32 GB GPUs start at 1200 € and then it goes further up, in model quality and price. (Top models require something like 60- 200 GB of GPU memory I think) So for sure any medium sized company could afford to run their own LLMs, also at scale if they want to make the investment. The question is, how much they value their confidential data. (I would not trust any of the big AI companies). And you don't usually need cutting edge reasoning and coding abilities to process basic information.
- FuckButtons 11mo agoA 120gb ram MacBook Pro will run gpt-oss-120b at a very respectable clip and I’ve found it to be quite serviceable for a lot of tasks.
- adastra22 11mo agoI bought one for this purpose, but LM Studio doesn't seem to want to run even the most quantized versions. Any suggestions?
- 11mo ago
- _joel 11mo agoYou can get local RAG with Anythingllm if you want minimal effort too fwiw. Pretty much plug and play. Used it for simple testing for an idea before getting into the weeds of langchain and agentic RAG.
- kbrisso 11mo agoI built this for local RAG https://github.com/kbrisso/byte-vision https://github.com/kbrisso/byte-vision it uses llama.cpp and Elasticsearch. On a laptop with 8 GB GPU it can handle a 30K token size and summarize a fairly large PDF.
- urbandw311er 11mo agoWhen it comes to the evals for this kind of thing, is there a standard set of test data out there that one can work with to benchmark against? ie a collection of documents with questions that should result in particular documents or chunks being cited as the most relevant match.
- autogn0me 11mo agoYes check out haiku-rag benchmarks and evaluations
- dwa3592 11mo agoIf you end up using any of the frontier models, don't forget to protect private information in your prompts - https://github.com/deepanwadhwa/zink https://github.com/deepanwadhwa/zink
- cjonas 11mo agoDoesn't seems necessary if you are using claude via bedrock or gpt via azure. At that point, its not different then sending PII through a serverless function.
- wanderingmind 11mo agoCare to explain more? I understand the prompt might not be used for training, but how about sanitizing the PII from tracking or logging or memory bugs in these serverless functions
- cjonas 10mo agoMy point is there are plenty of cases where you would send the same PII through a server-less function or internal API (IE PATCH /user/profile). The concerns about logging or bugs are the same in both instances. You could make a case that using a masking tool like this would make it easier to share full production logs, but there are plenty of other ways to secure logs that don't involve modifying the runtime behavior.
- mijoharas 11mo agoI'm interested in the embeddings models suggested. I had some good results with nomic in a small embedding based tool I built. I also heard a few good things about qwen3-embedding, though the latency wasn't great for my usecase so I didn't pursue it much further. Similarly, I used sqlite-vec, and was very happy with it. (if I were already using postgres I'd have gone with that, but this was more of a cli tool). If the author is here, did you try any of those models? how would you compare the ones you did use?
- dmezzetti 11mo agoGlad to see all the interest in the local RAG space, it's been something I've been pushing for a while. I just put this example together today: https://gist.github.com/davidmezzetti/d2854ed82f2d0665ec7efdd073d575d7 https://gist.github.com/davidmezzetti/d2854ed82f2d0665ec7efd...
- johnebgd 11mo agoInteresting stack. I’ve been working on doing something like this with Apple specific tech. Swiftdata is not easy to work with.
- 0xC45 11mo agoFor an open source, local (or cloud) vector DB, I would also recommend checking out Chroma (https://trychroma.com https://trychroma.com). It also supports full text search. Disclaimer: I work on Chroma cloud.
- JKCalhoun 11mo agoI kinda do want to build a local RAG? I want some significant subset of Wikipedia (I assume most people know about these) on a dedicated machine with a RAG front-end. I would have then an offline Wikipedia "librarian" I could query. But I'm lazy and assumed that someone has already built such a thing. I'm just not aware of this "Wikipedia-RAG-in-a-box".
- yakkomajuri 11mo agoIn this case do you even need a RAG? Most models will have been trained on Wikipedia anyway. Give Jan (https://www.jan.ai/ https://www.jan.ai/) a try for instance. You'll need to do a bit of research as to what model will give you the best perf on your system but one of the quantized Llama or Qwen models will probably suit you well.
- JKCalhoun 11mo agoThank you.
- jonwinstanley 11mo agoStandard Ollama probably covers a lot of this
- ElasticBottle 11mo agoHow does this compare with orama?
- autogn0me 11mo agoWhat we use: - https://github.com/ggozad/haiku.rag https://github.com/ggozad/haiku.rag Why? - developer oriented (easy to read Python and uses pydantic-ai) - benchmarks available - docling with advanced citations (on branch) - supports deep research agent - real open source by long term committed developer not fly by night
- into_the_void 11mo agoInteresting perspective on the use of full-text search over vector databases for RAG. I appreciate the insights on agentic tool loops and handling fuzzy searching.
- andai 11mo agoWhen I started playing with this stuff in the GPT-4 days (8K context!), I wrote a script that would search for a relevant passage in a book, by shoving the whole book into GPT-4, in roughly context sized chunks. I think it was like a dollar per search or something in those days. We've come a long way! Anthropic, in their RAG article, actually say that if your thing fits in context, you should probably just put it there instead of using RAG. I don't know where the optimal cutoff is though, since quality does suffer with long contexts. (Not to mention price and speed.) https://www.anthropic.com/engineering/contextual-retrieval https://www.anthropic.com/engineering/contextual-retrieval The context size and pricing has come so far! Now the whole book fits in context, and it's like 1 cent to put the whole thing in context. (Well, a little more with Anthropic's models ;)
- adastra22 11mo agoRust API?
- yakkomajuri 11mo agoPost author (not OP) here. Would you want a Rust SDK for Skald?
- adastra22 11mo agoYes, none of the languages listed are very accessible from the frameworks I'm working with.
- yakkomajuri 11mo agoEmailing you!
- davedx 11mo ago> we use Sentence Transformers (all-MiniLM-L6-v2) as our default (solid all-around performer for speed and retrieval, English-only). Huh, interesting. I might be building a German-language RAG at some point in my future and I never even considered that some models might not support German at all. Does anyone have any experience here? Do many models underperform or not support non-English languages?
- architectonic 11mo agoYes I can confirm that,we had resorted to a multilingual embedding model back in the day. https://link.springer.com/chapter/10.1007/978-3-031-77918-3_15 https://link.springer.com/chapter/10.1007/978-3-031-77918-3_...
- navar 11mo agoYou can refer to https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard and use that to guide your selection. Check under the "Retrieval" section, either RTEB Multilingual or RTEB German (under language specific). You may also want to filter for model sizes (under "Advanced Model Filters"). For instance if you are self-hosting and running on a CPU it may make sense to limit to something like <=100M parameters models.
- davedx 11mo agoThanks, that's really useful, I had no idea this table existed.
- yakkomajuri 11mo ago> Do many models underperform or not support non-English languages? Yes they do. However: 1. German is one of the more common languages to train on so more models will support it than say, Bahasa 2. There should still be a reasonable amount of multi-lingual models available. Particularly if you're OK with using proprietary models via API. AFAIK all the frontier embedding and reranking models (non open-source) are multi-lingual
- Oras 11mo agoThe hardest part in RAQ is document parsing. If you only consider text then it should be ok, but once you start having tables, tables going multiple pages, charts, ignore TOC when available, footnotes … etc, that part becomes really hard and accuracy suffers to get the context regardless of what chunking do you use. There are some patterns to help such as RAPTOR where you make ingestion content aware and instead of just ingesting content, you start using LLMs to question and summarise the content and save that to the vector database. But reality is, having one size fits all for RAQ is not an easy task.
- Royce-CMR 11mo agoSuper noob in vector embeddings: I never considered that tables would be a complexifier. (beyond defining in a parseable format for ingestion). Do vector databases do better with long grouped text vs table formats?
- Oras 11mo agoThe issue is the ingestion (extracting the right data in the right format). This is mainly an issue in PDFs and sometimes when there are tables added as images in Docx too. You need a mix of text and OCR extraction to get the data correctly first before start chunking and adding embeddings
- nh2 11mo agoI'd like to have a local, fully offline and open-source software into which I can dump all our Emails, Slack, Gdrive contents, Code, and Wiki, and then query it with free form questions such as "with which customers did we discuss feature X?", producing references to the original sources. What are my options? I want to avoid building my own or customising a lot. Ideally it would also recommend which models work well and have good defaults for those.
- cbcoutinho 11mo agoThis is why I built the Nextcloud MCP server, so that you can talk with your own data. Obviously this is Nextcloud-specific, but if you're using it already then this is possible now. https://github.com/cbcoutinho/nextcloud-mcp-server https://github.com/cbcoutinho/nextcloud-mcp-server The default MCP server deployment supports simple CRUD operations on your data, but if you enable vector search the MCP server will begin embedding docs/notes/etc. Currently ollama and openai are supporting embeddings providers. The MCP server then exposes tools you can use to search your docs based on semantic search and/or bm25 (via qdrant fusion) as well as generate responses using MCP sampling. Importantly, rather than generating responses itself, the server relies on MCP sampling so that you can use any LLM/MCP client. This MCP sampling/RAG pattern is extremely powerful and it wouldn't surprise me if there was something open source that generalizes this across other data sources.
- russdill 11mo agoWould love to see someone build an example using the offline wikipedia text.
- throwaway19343 11mo agoNo
- spacecadet 11mo agoYou can vibe code a local RAG with or without vectors in 5 minutes. Like another commenter pointed out, unless your corpus is huge, you do not need vectors, but hey using vectors is fun so why not. For what its worth, I run a local first, small model, private RAG that uses LangGraph, Neo4J knowledge graphs, I swap the models around constantly. It mostly just gets called by agent tools now.
- okeuro49 11mo agoDo you run the models locally? No local model for me manages to get function calling right.
- spacecadet 11mo agoAre you using LangGraph tool nodes? I run very small, non-RLHF instruction models with maybe a 2% failure rate on response format matching tool definition. I would also guess you do not have your orchestration and pipe configured correctly.
- deleted 11mo ago[deleted]
- abhashanand1501 11mo agoMy advice - use same rigor as other software development for a RAG application. Have a test suite (of say 100 cases) which says for this question correct response is this. Use an LLM judge to score each of the outputs of the RAG system. Now iterate till you get a score of 85 or so. And every change of prompts and strategy triggers this check, and ensures that output of 85 is always maintained.
- mingodad 11mo agoI did an experiment while learning about LLMs and llama.cpp consisting in trying to use create a Lua extension to use llama.cpp API to enhance LLMs with agent/RAG written in Lua with simple code to learn the basics and after more than 5 hours chatting with https://aistudio.google.com/prompts/new_chat?model=gemini-3-pro-preview https://aistudio.google.com/prompts/new_chat?model=gemini-3-... (see the scrapped output of the whole session attached) I've got a lot far in terms of learning how to use an LLM to help develop/debug/learn about a topic (in this case agent/RAG with llama.cpp API using Lua). I'm posting it here just in case it can help others to see and comment/improve it (it was using around 100K tokens at the end and started getting noticeable slow but still very helpful). You can see the scrapped text for the whole seession here https://github.com/ggml-org/llama.cpp/discussions/17600 https://github.com/ggml-org/llama.cpp/discussions/17600