15 ms·

Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

• deleted 3y ago
  [deleted]
• ilaksh 3y ago
  You don't train on documents. There are many startups claiming that but they are deliberately using a misleading term because they know that's what people are searching for. You still do RAG. Llamaindex is still the best option that I know of. Most of the startups that have working products are likely using llamaindex. All of the ones that say they are training on documents are actually using RAG. Test it out. If it really and truly doesn't work, search for a script that creates question and answer pairs automatically with gpt-4. Then try using that for qLoRA. I have never heard of anyone successfully using that for a private document knowledgebase though. Only for skills like math, reasoning, Python, etc. I think the issue is that you need a LOT of data and it needs to repeat concepts or any facts you need to learn many, many times in different supporting ways. What absolutely does not work is trying to just feed a set of documents into fine tuning. I personally have proven that dozens of times because I had a client who is determined to do it. He has been mislead. What it will do is learn the patterns that are in those documents.
  • min76 3y ago
    Well said. The problem is, there are way too many alternatives. Any idea how llamaindex's ingestion engine compares to unstructured.io? ( Which is used in langchain)
    • ilaksh 3y ago
      I think they may be using the same thing.
  • dfhg 3y ago
    Are there public examples of working products using RAG, compared with fine-tuning or training from scratch?
    • c0pium 3y ago
      Copilots use RAG: https://www.microsoft.com/en-us/research/group/dynamics-insights-apps-artificial-intelligence-machine-learning/articles/prompt-engineering-improving-our-ability-to-communicate-with-an-llm/ https://www.microsoft.com/en-us/research/group/dynamics-insi...
    • tinco 3y ago
      The OpenAI assistants API is an implementation of a RAG pipeline. It performs both RAG on any documents you upload, and on any conversation you have with it that exceeds the context.
    • regularfry 3y ago
      Amazon Q is (at least partially) a RAG implementation.
    • aik 3y ago
      Not public but internally I wrote a tool to help us respond to RFPs. You pass in a question from a new RFP and it outputs surprisingly great answers most of the time. Is writing 75%+ of our RFP responses now (naturally we review and adjust sometimes and as needed). And best of all it was very quickly hacked together and it’s actually useful. Copied questions/answers from all previous ones into a doc, and am using OpenAI embeddings api + FAISS vector db + GPT-4 to load the chunks + store the embeddings + process the resulting chunks.
      • deleted 3y ago
        [deleted]
  • dfhg 3y ago
    Another question, which one is preferred, LlamaIndex or Langchain, for RAG? Thanks in advance for your insights.
    • isoprophlex 3y ago
      You basically don't use langchain for anything besides 30 minute demos that you copied from someone else's github. It has a completely spaghettified API, is not performant, and forces you into excessive mental contortions to reason about otherwise simple tasks. LlamaIndex is pretty good.
      • monkeydust 3y ago
        Yea discovered this with Langchain last week. Was great for a demo then started to push it harder and spent ages trawling Reddit, discord, GitHub trying to find solutions to issues only to discover what was supposed to be supported was deprecated. Got a massive headache for what should have been a simple change. Moved on now.
      • sophiabits 3y ago
        Yeah +1 We originally started out building features with LangChain (loading chains from YAML sounded good—it felt like it would be easy to get non-engineers to help with prompt development) but in practice it’s just way too complicated. Nice idea, but the execution feels lacking. It also doesn’t help that LangChain is evolving so rapidly. When we first started using it a lot of code samples on the internet couldn’t be copy/pasted because of import paths changing, and at one point we had to bump by ~60 patch versions to get a bug fix, which was painful because it broke all kinds of stuff
    • simonw 3y ago
      LlamaIndex is mainly focused on RAG. LangChain does a ton of other stuff too. I'd focus on LlamaIndex first.
    • mathis-l 3y ago
      Haystack [1] is another good option. It‘s modular, doesn’t get in your way and is particularly strong at retrieval. People like the documentation too. Disclaimer: I work at deepset [1] https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack
  • benjaminwootton 3y ago
    To sing the praises of Bedrock again, it does have continuous pre-training as well as RAG “knowledge bases”. The former is based on JSON fragments and the RAG stuff is PDFs and other document formats. With regards to its efficacy, I haven’t gone to production with it yet but I was reasonably impressed. I uploaded 100 legal case documents to Bedrock via Claude and could push it pretty hard asking about the various cases and for situations across the knowledge base. It did feel like it broke down and got confused at a certain point of complexity of questioning, but I still think it’s already useful as a “copilot” or search engine and surely it will only improve over time.
    • ilaksh 3y ago
      I forgot about the continuous pre-training thing. How long and how much did they cost on Bedrock? I had tried to suggest continuous pre-training to my client but it seemed expensive and when I mentioned that he lost interest and just kept wanting me to do fine tuning. Also to clarify, did you do the continuous pre-training or RAG? And did you compare the efficacy of one or the other or both?
      • benjaminwootton 3y ago
        I used the RAG knowledge bases for most of my testing described above. I got a toy demo up and running with continuous pre-training but haven’t evaluated it unfortunately.
    • hbamoria 3y ago
      Oh Great! How did you evaluate the LLM responses? I'm cofounder of an evaluation and monitoring platform - Athina AI (www.athina.ai) You can use our monitoring dashboard and evals to check your LLM performance and iterate quickly.
  • seedless-sensat 3y ago
    What is RAG? That's hard to search for
    • deleted 3y ago
      [deleted]
    • accrual 3y ago
      Retrieval-augmented generation, RAG + LLM will turn up more results.
    • rjzzleep 3y ago
      This one seems like a good summary Retrieval-Augmented Generation for Large Language Models: A Survey https://arxiv.org/abs/2312.10997 https://arxiv.org/abs/2312.10997 The photos of this post are also good for a high level look https://twitter.com/dotey/status/1738400607336120573/photo/2 https://twitter.com/dotey/status/1738400607336120573/photo/2 From the various posts I have seen people claim that phi-2 is a good model to start off from. If you just want to do embeddings, there are various tutorials to use pgvector for that.
    • nmstoker 3y ago
      Seems fairly easy to search for to me - top results are all relevant: https://kagi.com/search?q=ml+rag https://kagi.com/search?q=ml+rag https://www.google.com/search?q=ml+rag https://www.google.com/search?q=ml+rag
    • tomduncalf 3y ago
      Retrieval Augmented Generation - in brief, using some kind of search to find relevant documents to the user’s question (often vector DB search, which can search by “meaning”, by also other forms of more traditional search), then injecting those into the prompt to the LLM alongside the question, so it hopefully has facts to refer to (and its “generation” can be “augmented” by documents you’ve “retrieved”, I guess!)
      • whartung 3y ago
        So, as a contrived example, with RAG you make some queries, in some format, like “Who is Sauron?” And then start feeding in what books he’s mentioned in, paragraphs describing him from Tolkien books, things he has done. Then you start making more specific queries? How old is he, how tall is he, etc. And the game is you run a “questionnaire AI” that can look at a blob of text, and you ask it “what kind of questions might this paragraph answer”, and then turn around and feed those questions and text back into the system. Is that a 30,000 foot view really of how this works?
  • sroecker 3y ago
    We just held a workshop about this a few weeks ago: https://red.ht/llmappdev https://red.ht/llmappdev We created a simple chatbot using local models with Ollama (llamacpp), LlamaIndex and streamlit. Have a look at the streamlit folder, it's super easy. I used this simple example to teach about RAG, the importance of the system prompt and prompt injection. The notebook folder has a few more examples, local models can even do natural language SQL querying now.
    • 3abiton 3y ago
      Llamaindex has so mucu potential. Any benchmarks on performance compared to fine-tuning?
      • sroecker 3y ago
        You probably don't need fine-tuning, at least if it's just new content (and no new instructions). It may even be detrimental, since LLMs are als good at forgetting: https://twitter.com/abacaj/status/1739015011748499772 https://twitter.com/abacaj/status/1739015011748499772
    • agilob 3y ago
      looks very promising, do you plan to keep this single repo up to date as new things are released?
      • sroecker 3y ago
        Good question, as you can see I haven't touched it for a month. I wanted to show what's possible then with open source and (open) local models and there's already so much new stuff out there. I'll probably fix some things this week and then either update it or start from scratch. Guided generation, structured extraction, function calling and multi-modal are things I wanted to add and chainlit looks interesting.
  • spacecadet 3y ago
    Ouch your client! I had one earlier this year like this. We were doing some audio processing for word matching, he had also been mislead before coming to us, he fully believed that this was going to be some form of super AI trained on his 5 audio records of him repeating the words over and over... We did all we could to steer him toward a correct path of understanding. Sadly we launched a working product but he doesn't understand it and continues to miss represent and miss sell it. After continuing to give him time and follow up with him (I tend to personally do this with Clients like this), I can tell he is starting to realize his lack of understanding...
  • jasonjmcghee 3y ago
    You don't just feed documents in, you need to build a dataset representative of how you want to interact with it. So likely using gpt-4 or something to create: a chunk of a document, a question that can be answered by that chunk and a good answer. (Or something)
  • stevenhuang 3y ago
    > What absolutely does not work is trying to just feed a set of documents into fine tuning. Not quite. It does work, albeit likely not optimal. See https://github.com/bublint/ue5-llama-lora https://github.com/bublint/ue5-llama-lora
    • ilaksh 3y ago
      Have you tried that? I find the results hard to believe because he says he is using Llama 7b and asking it questions but that is not a chat model. Also he does not appear at all in the issues when people ask him about reproducing the results. Instead there are people in the issues recommending RAG or creating a QA dataset.
  • treprinum 3y ago
    LlamaIndex can't do chunk-level metadata, only document-level metadata, so you can't put precise references to where materials the LLM synthesized answers from originated, e.g. HTML anchors. Just write your own RAG with Pinecone and OpenAI APIs directly.
    • seldo 3y ago
      LlamaIndex lets you attach metadata to Nodes which are basically chunks, although that fact is poorly documented! Will fix.
      • treprinum 3y ago
        Thanks! Even with a better documentation, document importers don't extract node metadata so one needs to write their own "text and metadata extractor" as well. It's then easier to skip LlamaIndex altogether, or just get inspiration from some re-ranking etc. you guys did.
        • anjanb 3y ago
          can you elaborate please
  • jncfhnb 3y ago
    RAG is a funny thing. It’s like going back to Watson for specifics but letting the LLM handle the generic stuff.
  • thorum 3y ago
    Has anyone tried using an LLM for the retrieval stage? Instead of using vector embeddings, have a (small, fast) LLM literally scan the entire corpus in chunks extracting relevant sections?
    • spullara 3y ago
      That would still be very slow with any reasonably large corpus.
  • warthog 3y ago
    Why Llamaindex instead of Langchain?
  • moltar 3y ago
    Another super easy option for RAG is AWS Bedrock Knowledge Base. It can ingest docs from S3. Just don’t use the OpenSearch serverless store it’s $$$. Can use a low end RDS with pgvector extension.
  • legendofbrando 3y ago
    What’s the benefit of llamaindex over just storing documents in chroma and using chroma to query? I’ve done the latter and trying to understand if there’s a performance gain to the former?
    • sivasurend 3y ago
      Not much, actually. For lower volumes of documents, vector stores like Chroma or Weaviate provide inbuilt RAG. Things get messy when the number and type of documents increase. Below are the reasons why you may need advanced RAG. 1. Intelligent Data Parser 2. Chunking efficiently 3. Choice of embedding models 4. Query transformation 5. RAG technique 6. Prompt design 7. Feedback loop Check out my blog on the 27 parameters, considerations and techniques one could follow to build a State-of-the-Art Chatbot. https://www.lyzr.ai/27-parameters-techniques-considerations-for-building-a-llm-powered-state-of-the-art-sota-chatbot/ https://www.lyzr.ai/27-parameters-techniques-considerations-... So here is the quick guide. For simpler usecases - inbuilt vector database RAG is sufficient For more complex ones - LlamaIndex or Langchain options are suitable For enterprise grade production use cases - Lyzr's SOTA RAG architecture comes in handy
  • vikbehal 3y ago
    [dead]
• benjaminwootton 3y ago
  AWS Bedrock is fairly easy. You can do it in 5 or 6 clicks. You have to upload your documents to S3, create a “Knowledge Base” then sync your documents into a vector database like OpenSearch or PineCone. You are then good to go via their playground or the AWS API. I made a video here describing the process, check around 14 minutes in: https://ensembleanalytics.io/blog/introducing-bedrock-knowledge-bases https://ensembleanalytics.io/blog/introducing-bedrock-knowle... Bedrock is a decent product I think. All of the models in one place (apart from the big dogs from OpenAI) and a common API across them.
  • hrdwdmrbl 3y ago
    Is there a limit? Could I create a knowledge base with 10,000 documents? 100k? 1M?
    • 8organicbits 3y ago
      https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html https://docs.aws.amazon.com/bedrock/latest/userguide/quotas....
      • hrdwdmrbl 3y ago
        I’m sorry, I don’t understand those limits. It uses a lot of unfamiliar terms like “batch inference” and “modality”. I just want a nice UI that I can give my hard-drive to and then ask it questions.
        • deleted 3y ago
          [deleted]
        • nighthawk454 3y ago
          That’s probably unrealistic at this time
          • xuancanh 3y ago
• lolinder 3y ago
  I haven't personally tried this for anything serious yet, but to get the thread started: Cheshire Cat [0] looks promising. It's a framework for building AI assistants by providing it with documents that it stores as "memories" that can be retrieved later. I'm not sure how well it works yet, but it has an active community on Discord and seems to be developing rapidly. The main perk over the cloud options is that you can point it at any language model, including fully local—my local install pointed at my local Ollama running Mistral. [0] https://github.com/cheshire-cat-ai/core https://github.com/cheshire-cat-ai/core
  • ilaksh 3y ago
    But that's not training. That's RAG. They seem to be using qdrant which I believe is a vector store.
    • lolinder 3y ago
      They've updated the question to clarify that RAG counts, and as many have noted, properly "training" on a set of documents isn't really a thing.
• galacticaactual 3y ago
  Train on your own documents or analyze your own documents for answers? Very different things. For the first (fine tuning) follow “AI Jason” on YouTube. He has some great tutorials. For the second (RAG or similar), fire up a cloud VM with GPUs or use Ollama locally and read through the LlamaIndex docs on how to build a RAG pipeline.
  • dfhg 3y ago
    Would you kindly elaborate a little bit the difference between training on own documents vs analyzing documents for answers?
    • simonw 3y ago
      The word "training" implies creating a new model by fine-tuning an existing model on top of new documents. As several other comments in this thread have already indicated: this is almost always the wrong direction. Which is confusing because it's the direction everyone always assumes they should go in at first. The approaches that does work is surprisingly simple: take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get. This is known as RAG: Retrieval Augmented Generation. It's a very powerful approach.
      • codetrotter 3y ago
        > take the user's question, search for snippets of your documents that appear to be about that question, then paste all of those snippets into the prompt along with the user's question and see what answer you get. We use RAG at my job, but we don’t do any preprocessing on the message from the user, so the results are not always great for us. Do any of you have experience using a small local model just for extracting keywords from messages which you then use for the retrieval? And then feed the search result and your prompt into OpenAI or whatever as normal.
        • ilaksh 3y ago
• xnx 3y ago
  GPT-4 Turbo has a 128K (~300 pages) context window, which probably handles a lot of use cases which might have previously needed extra training/refinement.
  • mettamage 3y ago
    The chatgtp app says it has a context window of 4096 tokens (gpt 4). How do I get access to turbo?
    • SgtBastard 3y ago
      The gpt-4-1106-preview (aka gpt-4-turbo) is a foundational model with a 128K context window that OpenAI makes available for API consumption both directly and via Azure OpenAI Services. ChatGPT is a consumer facing service that wraps the GPT-4 foundational model but at some point will likely wrap gpt-4-turbo. Signing up for OpenAI API access or Azure OpenAI Services will grant you access to this model (with some rate-limits in place given its a preview model).
    • coder543 3y ago
      If you're asking ChatGPT about its own characteristics, you should not believe the responses. Models can't examine themselves, so unless the model was trained on specific information about itself, or unless the info is put into the System Prompt, then it cannot know the answer. A response indicating 4096 tokens would just be a hallucination, or a misunderstanding based on training data that saw references to what older versions of ChatGPT were trained on. ChatGPT-4 is powered by GPT-4 Turbo at this point, so it has a context window that is much larger than 4096 tokens, whether it knows it or not. The ChatGPT application may limit the context size from reaching the full 128k to keep costs down, and it will be using some of the context window for its own purposes, but it's certainly able to maintain context across more than 4096 tokens of conversation.
    • deleted 3y ago
      [deleted]
  • Der_Einzige 3y ago
    Long context length models are still mostly a mirage with the "lost in the middle" phenomenon rearing it's ugly little head on actual production use-cases of this.
• tucared 3y ago
  Hi all, I was asking myself the same thing today and I followed a recent blog post from LlamaIndex[0] to create the following repo https://github.com/tucared/llm-file-explorer https://github.com/tucared/llm-file-explorer [0] https://blog.llamaindex.ai/running-mixtral-8x7-locally-with-llamaindex-e6cebeabe0ab?gi=831e3e521f6c https://blog.llamaindex.ai/running-mixtral-8x7-locally-with-...
• kmkarim 3y ago
  Slightly off topic but is there recommended advice on how to tune / train not for document retrieval but for consistent JSON output with specific enums? i.e given a text, always return back a certain set of fields. For some keys here is the possible set of enums etc. One shot prompting does work but curious how others approach this if you have training data on hand.
  • techn00 3y ago
    You want grammars to restrict the output, search for "gbnf grammar". That and combined with a good prompt with an example, also check out outlines.dev
  • Hugsun 3y ago
    There are many interesting tools that achieve this, like Outlines[0] and jsonformer[1]. I haven't tried them myself but they look very promising. [0]: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines [1]: https://github.com/1rgs/jsonformer https://github.com/1rgs/jsonformer
  • simonw 3y ago
    For OpenAI, use their functions schema mechanism. Aside from that, take a look at llama.cpp grammars.
  • ukuina 3y ago
    Microsoft Guidance will do this.
    • Der_Einzige 3y ago
      Seems Microsoft spun them out and gave them independence. Not sure why, given it's the kind of IP that helps keep microsoft dominant.
  • michaelt 3y ago
    Ask the model nicely, check any json in the output against your schema, regenerate if it doesn’t match. Crude, I know, but it’s compatible with every model. Which is useful if you want to compare the many different models out there.
• jrpt 3y ago
  What are you trying to do more specifically? You can use https://docalysis.com/ https://docalysis.com/ for most document RAG tasks.
• quickthrower2 3y ago
  Easiest is OpenAI assistants api. Use the playground and it’s a no code experience.
  • altdataseller 3y ago
    How do you upload the documents? Via the API or do you have to upload them beforehand through the UI?
    • footlose_3815 3y ago
      You upload in the GUI. It's available on free accounts as well, as long as you use GPT 3.5. There's a dropdown. It's super-convenient. I uploaded an AWS study guide, and I'm asking for example questions for my testing. As far as I can see, I can tell no way to determine if it's pulling from the guide, or if it's from GPT's data.
• sophiebits 3y ago
  If you’re looking for something that is hosted for you, at Notion we launched a feature for this a few weeks ago and it works quite well in my experience. RAG is one of the techniques used. https://www.notion.so/blog/introducing-q-and-a https://www.notion.so/blog/introducing-q-and-a
  • wanderingbit 3y ago
    Thank you! I have been reading these QLoRa posts in the hopes of training it on my notes stored in Notion, but then you do it for me! Nice product ;).
• constantinum 3y ago
  Unstract - https://unstract.com/ https://unstract.com/ They are a month away from launch(both open source and cloud) The team might be able to give you a quick demo on your specific requirements.
• iAkashPaul 3y ago
  A go-to method is to ingest different chunksizes based on the document hierarchy & then use langchain with a bunch of retrievers depending on the doc type. Then create an index about the metadata of each doc. So that you can ask the RAGbot what all it can answer about. Another way to ensure it stays on-domain is to generate synthetic questions & check for similarity against user queries. There's a whole rabbit hole of query decomposition to avoid straying off topic as well.
• csbartus 3y ago
  https://khoj.dev/ https://khoj.dev/ Tried this summer, and kinda worked!
• liampulles 3y ago
  What is your usecase? If you want to search for relevant info in your documents and get relevant info, and you want to avoid hallucination, you might avoid the text generation altogether. Instead you can extract text embeddings from your documents, put them in a vector DB, and then you have a super search. You can convert your search query to an embedding, search the DB and keep the e.g. 10 closest matches.
• viraptor 3y ago
  So far the recommendations are mostly hosted, so here's one local: https://github.com/weaviate/Verba https://github.com/weaviate/Verba I'm very happy with its results, even though the system is still young and a little bit janky. You can use it with either GPT API, or your local models through LiteLlm. (I'm running ollama + dolphin-mixtral)
  • ParetoOptimal 3y ago
    Thanks, I'd rather not feed the borgs.
• ukuina 3y ago
  PrivateGPT is one of the better-known examples, but most people are not aware that GPT4 Assistants handle RAG natively now: https://platform.openai.com/docs/assistants/overview https://platform.openai.com/docs/assistants/overview
• yu3zhou4 3y ago
  And what’s the correct answer in December 2023 if one wants to narrow down only to tools and services provided on Azure?
  • yu3zhou4 3y ago
    Is Llamaindex + hosted model on Azure OpenAI Services still the best option?
    • isoprophlex 3y ago
      There's Azure AI studio which is kinda like AWS bedrock. It's not bad, but for max control and versatility i'd start out rolling my own w/ for example llamaindex+azure-branded openai like you say.
      • yu3zhou4 3y ago
        Thanks. Is it possible to have persistent RAG in Azure AI Studio though? I found only a preview version of uploading files that are available to model, but when using this model through API, this uploaded data is not available to the model Likely I misunderstood about how RAG works with Azure AI Studio, so sorry in advance
        • isoprophlex 3y ago
          https://github.com/azure/aistudio-copilot-sample https://github.com/azure/aistudio-copilot-sample Check it out, specifically steps 3 and 4. As with almost every Microsoft CLI tool and SDK, it's clunky... and you can tell everyone is rushing this AI shit out as fast as they can to stay in the game. But what you want should be doable.
          • yu3zhou4 3y ago
            Much thanks to you
• GabrieleR 3y ago
  Try with private gpt github repo
• monkeydust 3y ago
  Did this in the summer via RAG. One thing we realised is that pure vector embeddings retrieval doesn't work so well for docs with acronyms (which let's face it all businesses have). Created a hybrid solution using embeddings and BM25 which is traditional ranking tool. This hybrid gave best results.
  • gardnr 3y ago
    I was going to ask how you integrated BM25 but then I found this: https://docs.llamaindex.ai/en/stable/examples/retrievers/bm25_retriever.html https://docs.llamaindex.ai/en/stable/examples/retrievers/bm2...
• throwaway421967 3y ago
  A bit unrelated, but one could open any binary file as text. With enough training data, could an llm just learn the format?
• drakonka 3y ago
  As mentioned above, I don't think you'd need to train your own model for this (or for most use cases of this, anyway). You'd use a RAG. I've tried out working with custom documents in two different ways for different types of data: * Once using LlamaIndex + Chroma[0] to transcribe and then conversationally query video contents (using GPT 3.5 or 4 as the backing LLM). * Once using GPT Plus, uploading long-form PDFs of my own fiction books to the GPT's knowledge base. I use this to help me remember character names and timelines (not always accurate, so results need to be treated with caution) and help brainstorm story or tech ideas for my world. Both work for what I'm using them for. I feel like option one is more customizable and easier to tweak for the types of results I would want, if I have concrete requirements about what kind of output I'm looking for. Option two has a lower barrier to entry and is just a little lower effort (no need to run your own app). For the next iteration, I'd like to try out AWS Bedrock and compare the workflow and results. [0] https://www.daily.co/blog/search-your-video-content-library-with-llamaindex-and-chroma/ https://www.daily.co/blog/search-your-video-content-library-...
• NKosmatos 3y ago
  There was something similar about Retrieval Augmented Generation (RAG) recently on HN: https://news.ycombinator.com/item?id=38491251 https://news.ycombinator.com/item?id=38491251 Early next year I’m preparing something similar for my team, so I’ll surely look into the useful links/recommendations posted by fellow HNers :-)
• cf1241290841 3y ago
  Show HN from two weeks ago mentioned this. https://news.ycombinator.com/item?id=38587052 https://news.ycombinator.com/item?id=38587052
• ignoramous 3y ago
  Here's a (video) guide on fine-tuning Mistral 7B with QLoRA: https://www.harpercarroll.com/articles/ai/llm-finetune-own-data https://www.harpercarroll.com/articles/ai/llm-finetune-own-d... / https://ghostarchive.org/varchive/kmkcNVvEz-k https://ghostarchive.org/varchive/kmkcNVvEz-k Fine tuning does result in degradation of the overall model (https://twitter.com/xaiguydotagi/status/1737082280835703142 https://twitter.com/xaiguydotagi/status/1737082280835703142) and so various RAG techniques may be desirable. As others have mentioned, LlamaIndex is a neat solution to build RAG pipelines: https://docs.llamaindex.ai/en/stable/optimizing/production_rag.html https://docs.llamaindex.ai/en/stable/optimizing/production_r...
  • jasonjmcghee 3y ago
    I strongly agree this is the direction the author is looking for. RAG is one approach, but if the query doesn't match the right documents, you're screwed. And often they use a different, much simpler, embedding model. I think harpercarrol link is a pretty good one, but it basically just feeds in the documents for completion, which isn't a good approach. The dataset needs to represent how you want to use it. This one might also be helpful https://www.deeplearning.ai/short-courses/finetuning-large-language-models/ https://www.deeplearning.ai/short-courses/finetuning-large-l... Honestly surprised how almost everyone is saying to use RAG (on its own). One strong benefit to RAG is the data can change, but has lots of failure modes. People often use hybrid search (fuzzy or bm25 etc alongside embedding search) which I suppose is still RAG. But fine-tuning models to be better at RAG is valuable as well, increasing accuracy. https://ragntune.com/blog/Fine-tuning-an-LLM-to-be-good-at-RAG https://ragntune.com/blog/Fine-tuning-an-LLM-to-be-good-at-R... Ideally, I'd try both. Fine tune on both the documents (create a question / answer dataset with gpt4) and rag instruction fine tune it.
• ndr_ 3y ago
  With OpenAI, you can first build Question & Answer pairs derived from your documents and use the OpenAI fine-tuning feature to build yourself a custom model. This method is more than just learning behavior in that facts do get recalled. I have written about it here, with a play demo use-case: https://ndurner.github.io/training-own-model-finetuning https://ndurner.github.io/training-own-model-finetuning Note that I have yet to use this in a real world use-case, and I would love to hear feedback. Other than OpenAI, there is the newly introduced „continued pre-training“ of Amazon Bedrock - but I haven’t tried. RAG: I think that‘s a fundamentally flawed concept, but RAGfluencers will disagree. ;-)
  • Akashic101 3y ago
    Could you expand on why RAG is flawed in your eyes? From the other answers it seems like its the way to go
• novaRom 3y ago
  We have to add LLMs and MMMs (multi modal models) into all standard Linux distributions. A service will index all local files creating embedding connectors, this will be used to augment user prompts, and voila we can search for anything with natural language.
  • btbuildem 3y ago
    Once models / inference engines are performant enough to run on consumer hardware without hogging all the resources of a machine -- sure! Embedding this into the core of a linux system sounds very interesting. A shell interaction could be had in nerd gobbledygook or plain human language.
  • pushfoo 3y ago
    > We have to add LLMs and MMMs (multi modal models) into all standard Linux distributions. Why? This seems like a quick road to remote-execution-vuln-as-system-service. > and voila we can search for anything with natural language. What if your model keeps misunderstanding you?
• ankit219 3y ago
  I think the answer depends on how many documents you have. To think in terms of tokens (assuming 750-1000 tokens is a page), if you have a good estimate of number of pages you want to query on, you can decide on the approach. Three popular approaches: 1. RAG: Most popular and works really well on smaller datasets. It is limited by number of vectors/embeddings. A typical embedding could be of 1000 tokens in size. Llamaindex did a lot of engineering on this and their techniques work pretty well. The problem with large datasets is almost always that users don't like writing long prompts/queries so the answers are more generic. 2. Finetuning + RAG: You can finetune a model on the expected outputs. If your datasets have the knowledge which might already be on open internet (blogposts, articles, anything non proprietary), then finetuning would work really well in combination with RAG, especially for large datasets. It may not work if you are working on a proprietary knowledge hard to find on open internet. 3. Continual pretraining - large large datasets, and when the knowledge is proprietary. I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Needs a model that is trained on their data and then Instruction Tuning of top of that. Most likely you wont need to do this
  • throwup238 3y ago
    > I talked to a firm with 70GB worth of data. No way a RAG pipeline would give them results. They are struggling to get LLMs to work for them. Wow so RAG is basically a toy for demos and low effort MVPs. 70GB is tiny, it’d barely qualify as “big data” 20 years ago. Is anyone trying more advanced stuff like knowledge graph augmented generation to try to expand on that?
    • hrdwdmrbl 3y ago
      I have a _small_ e-commerce company and we have >300GB. Most of that bulk is photos and videos though, but in an ideal world I’d like my AI assistant to find that stuff too: “I’m making a Boxing Day ad campaign. Can you show me the ads that we’ve made in previous years and all of the photos that we’ve taken of our new Reindeer and Elf designs?”
      • 3y ago
• CrypticShift 3y ago
  Here are ways to do it by simply adding files to an online interface. I mention them only because they are quite straightforward (and free) to set up. - https://notebooklm.google/ https://notebooklm.google/ (US or VPN): uses the "gemini pro" model. - poe.com: You need to "create a new bot", disable "Make bot publicly accessible," and then "add a knowledge source." this offers many models, although the best ones require a subscription.
• tesdinger 3y ago
  You can not get a non-hallucinating AI in 2023.
  • epolanski 3y ago
    Could you expand on the reasons and possibly relevant papers?
  • ParetoOptimal 3y ago
    It seems like there's not much point to LLM+RAG over a good local search engine. At least, that's what I've rapidly been concluding. I'm still holding out a bit of hope for there being more though.
• d4rkp4ttern 3y ago
  Many services/platforms are careless/disingenuous when they claim they “train” on your documents, where they actually mean they do RAG. An under-appreciate benefit of RAG is the ability to have the LLM cite sources for its answers (which are in principle automatically/manually verifiable). You lose this citation ability when you finetune on your documents. In Langroid (the Multi-Agent framework from ex-CMU/UW-Madison researchers) https://github.com/langroid/langroid https://github.com/langroid/langroid we’ve implemented a number of RAG techniques: our DocChatAgent uses a combination of lexical and semantic retrieval, reranking and relevance extraction to improve precision and recall: https://github.com/langroid/langroid/blob/main/langroid/agent/special/doc_chat_agent.py https://github.com/langroid/langroid/blob/main/langroid/agen... All the code is laid out clearly so can be tweaked. We have companies using Langroid in production (e.g. for customer-support); they especially like the RAG and multi-agent features. One of the interesting techniques in Langroid is a numbering trick for the relevance extraction stage (having the LLM extract verbatim relevant parts of passages) — instead of having the LLM “parrot” out the relevant portions, thus wasting time and tokens, we have it just spit out the relevant sentence numbers from a pre-annotated passage. We use a tool/function-call for this, leveraging Langroid’s task loop that seamlessly works for tool handling as well as sub-task handoff. Many interesting RAG applications often require more than simple question-answering (e.g extract structured info, match doc against requirements etc) and these scenarios benefit immensely from having multiple agents (so you get separation of concerns, modularity and easier state management). Langroid simplifies this type of multi-agent setup with its unique conversational task loops, e.g https://langroid.github.io/langroid/examples/agent-tree/ https://langroid.github.io/langroid/examples/agent-tree/ Colab quick start that builds up to a 2-agent system for extracting structured info from a document: https://colab.research.google.com/github/langroid/langroid/blob/main/examples/Langroid_quick_start.ipynb https://colab.research.google.com/github/langroid/langroid/b... Among many other things, Langroid also has full support for the OpenAI Assistants API, so you could use the “built-in” RAG from this API, which is a convenience but is a black box, I.e you don’t know what retrieval algo it is using, how it is filling context, and the tokens consumed.
  • mark_l_watson 3y ago
    Thanks. I just read through your Colab examples notebook.
• ryanSrich 3y ago
  What would be nice is some type of box or device I connect to a computer. I then give it full access to the system. It trains itself on all of the data on that computer. The device is now a portable LLM.
• ZunarJ5 3y ago
  I'm a fan of Khoj. Been using it for months. https://github.com/khoj-ai/khoj https://github.com/khoj-ai/khoj
  • ParetoOptimal 3y ago
    I need to give Khoj a try again. I tried it on my org inbox on a whim, couldn't get useful results, and promptly forgot about it.
    • npamart 3y ago
      Nice pun
• bmgoau 3y ago
  Run https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT Then make ingest /path/to/folder/with/files Then chat to the LLM. Done. Docs: https://docs.privategpt.dev/overview/welcome/quickstart https://docs.privategpt.dev/overview/welcome/quickstart
  • hrdwdmrbl 3y ago
    Others have said that RAG is only good up to a few 100s of files, eh? And since this is based on IndexLlama which is RAG-based, this would have the same limitation, eh?
  • patja 3y ago
    I've tried LocalGPT, PrivateGPT, and H2OGPT. Have you been satisfied with the responses you get from PrivateGPT? When I tried it, it seemed very shallow/cursory in its responses. I saw much more detailed and complete responses when trying H2OGPT.
    • bmgoau 3y ago
      The models released over the last two weeks are much much better than the defaults. Try changing settings.yaml to use https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGU... .
• jerpint 3y ago
  If you’re looking for an open source RAG solution, try our library: https://www.github.com/jerpint/Buster https://www.github.com/jerpint/Buster
• lngnmn2 3y ago
  [dead]
• joeyrobert 3y ago
  Gpt4all is a local desktop app with a Python API that can be trained on your documents: https://gpt4all.io/ https://gpt4all.io/
• min76 3y ago
  I have a related question. I have a fair idea of the LLM ecosystem. (Thanks to this very nice blog called Emerging Architectures for LLM Applications). The problem is, there are way too options in each component. ( For E.g, too many vector store implementations, ingestion engines etc) What is the easiest way to get started? Primarily around RAG on my own pdf files. Also, what is the best/easiest option for hosting?. That blog lists vercel,streamlit, streamship and modal. I know vercel at a high level and found it very good. I am not well versed with javascript/typescript though. I believe the best option for UI generation is to use one of their templates.
• assafe 3y ago
  Hey, GPT Researcher shows exactly how to do that with RAG. See here https://github.com/assafelovic/gpt-researcher https://github.com/assafelovic/gpt-researcher
• _giorgio_ 3y ago
  If you want a simpler task, like training a mistral llama etc on your documents, to act as a document completer, how would you proceed instead? Probably much easier. Thanks
  • baobun 3y ago
    Markov chains, or, if you need to get fancy, Hidden Markov models?
    • _giorgio_ 3y ago
      Is this a joke? I'd like to fine-tune an existing model, let's say mistral, on a new dataset using (existing tools), I've seen that there are a lot of approaches, but none that has gained traction, there isn't a clear consensus...
      • baobun 3y ago
        Not a joke. Without more specifics, doesn't sound like LLMs are what you need/want.
        • _giorgio_ 3y ago
          Thanks for wasting my time. I ask about fine tuning llama or mixtral and he answers with a nonsense, telling me that I don't want what I want.
          • matsemann 3y ago
            No, you asked for a document completer, and got a suggestion. You're behaving like a typical XY problem asker, and getting angry when you don't get the answer you want. And they hardly "wasted your time", it took you maybe all of 10 seconds to read their comment trying to help you. What a weird behavior from you. https://xyproblem.info/ https://xyproblem.info/
• newstio 3y ago
  how to run a local llm model for RAG apps. Retrieval documents are turkish. But ı would to analyze this documents with llm. But ı have not a turkish local llm. How to solve this problem. Out of fine-tune and training.
  • ametrau 3y ago
    If the LLM you use supports Turkish (I am pretty sure chatgpt does) then the language doesn’t matter. Augment the Generation by Retrieving turkish documents / snippets.
    • newstio 3y ago
      Thank you. but I don't want to use gpt. I want to use local llm. LLM does'not support Turkish a good level.
• BeetleB 3y ago
  Since no one has mentioned it so far: I did just this recently with txtai in a few lines of code. https://neuml.github.io/txtai/ https://neuml.github.io/txtai/
• noiv 3y ago
  How you do RAG with embeddings NOT in English? I mean there are a few thousand more languages.
  • dbish 3y ago
    Just use your embeddings model of choice that works with your language, I believe Ada from openai is multilingual but I don’t know what languages it works well on, there are many embedding models out there, huggingface is your friend in this search. The output is just a vector and the rest of the system can basically stay the same. The only other thing that may need to change depending on language is any text preprocessing you need to do like word or sentence breaking for languages with compound words (German), agglutination (Turkish), etc.
    • noiv 3y ago
      Well, thanks, how do they deal with technical terms never seen during training?
      • dbish 3y ago
        Unknown words can’t really be handled other than mapping to a placeholder “unknown” vector in most cases. This can be dealt with in a variety of ways, text preprocessing being one of them
  • htrp 3y ago
    Embeddings are not just english based ....
• mrbonner 3y ago
  I'm curious about this as well but my data is mostly (95%) numerical metrics. Is there a "RAG" mechanism for numerical data instead of text? My use case is data analysis, insight discovery for example.
  • nkrumm 3y ago
    im not an expert, but take a look at the llamaindex structured data retrieval examples: https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/structured_data.html https://docs.llamaindex.ai/en/stable/understanding/putting_i... There are also examples that combine semantic retrieval and structured retrieval.
• 627467 3y ago
  I've been looking for the answer to this to have a chat interface to my obsidian markdown notes (the whole vault, not just rag of individual notes). Will be following these threads closely
• kkielhofner 3y ago
  As others have said you want RAG. The most feature complete implementation I've seen is h2ogpt[0] (not affiliated). The code is kind of a mess (most of the logic is in an ~8000 line python file) but it supports ingestion of everything from YouTube videos to docx, pdf, etc - either offline or from the web interface. It uses langchain and a ton of additional open source libraries under the hood. It can run directly on Linux, via docker, or with one-click installers for Mac and Windows. It has various model hosting implementations built in - transformers, exllama, llama.cpp as well as support for model serving frameworks like vLLM, HF TGI, etc or just OpenAI. You can also define your preferred embedding model along with various other parameters but I've found the out of box defaults to be pretty sane and usable. [0] - https://github.com/h2oai/h2ogpt https://github.com/h2oai/h2ogpt
  • asah 3y ago
    I just tried installing this on a fresh GCP instance with NVidia T4 GPU, and let's just say it was non-trivial. The CPU version running on my mac was mostly an OK install and worked pretty well.
• JacobiX 3y ago
  I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems? For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?
  • jxmorris12 3y ago
    Good embeddings are supposed to handle this natively. For example the embedding for API should be close to the embedding for Application Programming Interface. Purportedly.
  • vl 3y ago
    AFAIK, modern tokenizers/vocabs don’t have OOV anymore because they use sub-word tokens if there is no entry for the given word. This also works for languages with composite words. In your example sum of embeddings for non-typo and typo version should have close cosine distance if it trained correctly.
• elvyscruz 3y ago
  anything-llm looks pretty interesting and easy to use https://github.com/Mintplex-Labs/anything-llm https://github.com/Mintplex-Labs/anything-llm
• Igor_Wiwi 3y ago
  It's super easy, an example could be found here https://technoclub.bearblog.dev/creating-a-simple-ai-chat-with-your-own-pdf-application/ https://technoclub.bearblog.dev/creating-a-simple-ai-chat-wi...
• gsharma 3y ago
  I have usually seen people recommend to chunk by sentences or paragraphs or some fixed length of characters. IMO, all these are suggested because they are easy to write code for, but in reality, length of a meaningful chunk depends entirely on the data. The way we chunk an FAQ document vs a PRD is different. Based on this assumption, I have a couple of questions: 1. Is chunking the most significant factor in RAG quality? 2. If there are no limitations, would humans that are experts in that dataset, be the best people to create chunks?
• rplp 3y ago
  My approach was not to train the model on the documents, as others mentioned. I built a vector database from the documents, and I query the questions against it, which is very fast. This is the RAG (retrival augmented generation) step others mentioned. The results, which are literal extracts from the documents, but short ones, are given to the model, which produces an answer. This is the slow part. I used many of Langchain's tools to manage the whole process. You can try it on Supawiki, with one of the featured wikis. Then, if you are ok with a solution that hosts your documents for you, you can upload them and use our solution.
• agnosticmantis 3y ago
  https://notebooklm.google https://notebooklm.google
• orenlindsey 3y ago
  You can pay for a ChatGPT account and upload your own documents. I didn't do this myself but my dad uploaded 6 years of sermon transcripts from our church. It sounds exactly like the pastor.
• narag 3y ago
  I'd very much appreciate if someone could clarify what exactly is needed in terms of hardware and software to implement these suggestions. Would a five years old laptop work? Do you need a beefy GPU? Are you using some prepackaged software?
• d7y 3y ago
  Try https://github.com/SecureAI-Tools/SecureAI-Tools https://github.com/SecureAI-Tools/SecureAI-Tools -- it's an open-source application layer for Retrieval-Augmented Generation (RAG). It allows you to use any LLM -- you can use OpenAI APIs, or run models locally with Ollama.
• staranjeet 3y ago
  You can use embedchain[1] to connect various data sources and then get a RAG application running on your local and production very easily. Embedchain is an open source RAG framework and It follows a conventional but configurable approach. The conventional approach is suitable for software engineer where they may not be less familiar with AI. The configurable approach is suitable for ML engineer where they have sophisticated uses and would want to configure chunking, indexing and retrieval strategies. [1]: https://github.com/embedchain/embedchain https://github.com/embedchain/embedchain
  • d-even 3y ago
    Yes, this is most intuitive and easy to use framework for building RAG applications. Do give it a try!
• ohthehugemanate 3y ago
  NGL I think this one has passed the point on the tech maturityc curve where it makes sense to roll your own I played with MS office's copilot builder the other day and it's amazing. Point it at a set of base URLs, uploaded files, public or behind authentication. In literal seconds you have a copilot that can be embedded anywhere, including messengers. I gave it the root of Azure documentation, the root of red hat documentation? And the root of the ansible documentation and it's excellent. It uses MS's open source LLM copilot framework, and you can swap out models for an open source one (instead of GPT) if you like.
  • samstave 3y ago
    Can you do me an experiment. Feed it a subreddit. Post results.
• soultrees 3y ago
  Is there any open source front ends out there? I know of anything LLM but hoping to plug my own home built rag system into a nice front end.
• devilsAdv0cate 3y ago
  [dead]
• tomgs 3y ago
  We do just that at Flexor! We've built what we call an Unstructured Data Transformation Layer. Think about it like an assembly line from raw text to tables in your data warehouse. We don't use Llamaindex, we have our own (proprietary) piece of tech that does this. We can and have been outputting gold-standard tables on top of a lot of different types of context (legal docs, call transcripts, customer reviews, chat logs, emails, blog posts, social media posts, etc...) and looking to expand to more interesting domains soon. If anyone wants to hear more hit me up at tom [at] flexor [dot] ai (does this still work or are scrapers smart enough nowadays to just grep for this too lol)
• vikbehal 3y ago
  Hello, I am currently working on a demo use-case to generate documents, and I intend to feed a few documents as sample. E.g., Say a leasing documents. Such document will vary by leasing company, state, etc. so there is not 1 template. I do understand that I could create embeddings for each template and then use them to ask ChatGpt to generate documents where certain entities would change. I've setup a basic project but I am stuck at stage where I don't know how to tell ChatGpt that provided documents are sample and it needs to generate similar ones based on prompt engineering.
• fuzzfactor 3y ago
  I would think you really need to get into action almost non-stop, maybe a couple all-nighters, since 2024 is almost here ;) Well over 40 years ago I certainly wasn't working on language models with only 16 kilobytes of memory and a 1MHz microprocessor. No "high-level" or human-readable language anyway. OTOH I always attempted to use the electronics further toward the limit of what it could provide, compared to average, and once the resource limit was reached (which occurred fairly quickly with only 16k) then the entire effort concentrated on maximizing the amount of machine learning that could be accomplished by code that fit in the memory. There was no distraction preparing for more powerful hardware to come, it wasn't going to be coming during the time period needed. No real artifical intelligence evolved, and there was nothing "general" about it. The idea was to select & collect the desired most useful inferences from the raw data and make them available to the operator's natural intelligence for all the high-level decision-making. Definitely no room to have data in memory, since it would waste the space you need for more thoughtful code. You can't have the resulting factors building up in memory either so they had to go to external storage as they were generated. Naturally to be used later by completely different code which is geared to process the rudimentary findings in relation to new data, and present that to the operator in order to enhance their pattern recognition and decision-making efforts. I could only imagine what it would be like if memory came in megabytes rather than merely the lowly kilobytes. One thing I think might still be true today, whatever amount of memory you have, you should be able to handle so much raw data that it makes the amount of memory look insignificant. OTOH, if you can't highly leverage a naturally intelligent operator without some huge resource requirements, you might not be on the right track when it comes to maximizing hardware utilization. And then there's the concept of analog noise amplification. You really need to be careful that there is nothing wrong or unrelated in the fundamental data set you are using at the time. Starting with a raw input signal, each stage of amplification will increase the amount of noise proportionally, and depending on the number of layers of amplification, any noise can cascade into top prominence when the desired signal is unfortunately weak. But the same level of noise-in-place-of-signal does not go away when it is dwarfed by a strong signal, the noise is merely masked during the high-signal passages but remains a considerable component. Now when the raw data only gets one initial pass, anything that's missed the first time is lost forever, and if the missing nuance is something important that would seem like the type of thing that when a strong need is there, anything related to that nuance would be unreliable, incorrect, false, or downright hallucinatory if the performance was advanced enough. GIGO is still the name of the game so I would think if it's custom training you have to step up to the plate and take the good with the bad. You've got to laboriously handle all the training data yourself anyway, so might as well take the opportunity to seriously babysit that data thoroughly in advance like you would never be able to do if you were only dealing with somebody else's already trained model. Plus you can't usually take an adequately huge data set and in one pass remove all the undesired artifacts. And if one pass results in a processed dataset which can no longer be considered very huge at all, you've probaly lost too much valuable information and you may still not have eliminated all the undesirable noise. This is somewhat analogous to lossy digital data compression, but focused on retaining only the most prominent meaning that can be gleaned from the data. As opposed to plain compression which retains the most prominent data regardless of meaning. The more intelligently the raw input data is handled, the more realistically intelligent you can expect the final outcome to turn out.
• treeskirt33 3y ago
  https://notebooklm.google/ https://notebooklm.google/