5 ms·

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need bill

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from these companies with powerful models and tweak it to deal with your own need.

CharlesW 3y ago
Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?
- rafaelero 3y ago
  This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to offer a good answer. But I am very impressed with GPT-4's abilities in finding the correct answer just by reading those documents, so I think the only problem still enduring is the context window size.
  - CharlesW 3y ago
    > My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. That was my impression as well, which is why your comment was so interesting to me. Have you found tools/projects for the DAG approach that you'd recommend?
ZunarJ5 3y ago
https://github.com/khoj-ai/khoj https://github.com/khoj-ai/khoj This is the easiest I found, on here too.
- nico 3y ago
  The readme says it’s offline and doesn’t mention any LLMs Does it not use any LLMs, or does it have its own, or how how does it work? Thank you
  - ZunarJ5 3y ago
    You can set it up so that it downloads Llama for offline use in the UI. You do not have to do it manually.
clharman 3y ago
@rafaelero I'm working on a blogpost (https://colinharman.substack.com/ https://colinharman.substack.com/) to demonstrate this fact since I get a lot of tiresome questions like "why don't you just train instead of retrieving" Do you have any scripts you could share for the training/eval process? Would love to credit you in the post

CharlesW 3y ago

Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?

rafaelero 3y ago
This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to offer a good answer. But I am very impressed with GPT-4's abilities in finding the correct answer just by reading those documents, so I think the only problem still enduring is the context window size.
- CharlesW 3y ago
  > My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. That was my impression as well, which is why your comment was so interesting to me. Have you found tools/projects for the DAG approach that you'd recommend?

ZunarJ5 3y ago

https://github.com/khoj-ai/khoj https://github.com/khoj-ai/khoj This is the easiest I found, on here too.

nico 3y ago
The readme says it’s offline and doesn’t mention any LLMs Does it not use any LLMs, or does it have its own, or how how does it work? Thank you
- ZunarJ5 3y ago
  You can set it up so that it downloads Llama for offline use in the UI. You do not have to do it manually.

clharman 3y ago

@rafaelero I'm working on a blogpost (https://colinharman.substack.com/ https://colinharman.substack.com/) to demonstrate this fact since I get a lot of tiresome questions like "why don't you just train instead of retrieving" Do you have any scripts you could share for the training/eval process? Would love to credit you in the post