3 ms·
which model are you using? what size/quant/etc? thanks!
by exe34 2y ago
which model are you using? what size/quant/etc?
thanks!
- axpy906 2y agoAgree. Please provide more details on this setup or a link.
- deegles 2y agoJust try a few models on your machine? It takes seconds plus however long it takes to download the model.
- exe34 2y agoI would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.
- NortySpock 2y agoNew LLM Prompt: Write a reddit post as though you were a human, extolling how fast and intelligent and useful $THIS_LLM_VERSION is... Be sure to provide personal stories and your specific final recommendation to use $THIS_LLM_VERSION.
- rkwz 2y agoNot the parent, but I started using Llama 3.1 8b and it's very good. I'd say it's as good as or better than GPT 3.5 based on my usage. Some benchmarks: https://ai.meta.com/blog/meta-llama-3-1/ https://ai.meta.com/blog/meta-llama-3-1/ Looking forward to try other models like Qwen and Phi in near future.
- milleramp 2y agoI found it to not be as good in my case for code generation and suggestions. I am using a quantized version maybe that's the difference.
- smcleod 2y agoCome join us on Reddit’s /r/localllama. Great community for local LLMs.
- wongarsu 2y agoI'd be interested in other people's recommendations as well. Personally I'm mostly using openchat with q5_k_m quantization. OpenChat is imho one of the best 7B models, and while I could run bigger models at least for me they monopolize too many resources to keep them loaded all the time.