24 ms·
There's a bunch of LLaMA instruction-tuned variants on huggingface and new ones are coming out every day. I personally had the best experience with GPT4-x-vicu
by alpark3 3y ago
There's a bunch of LLaMA instruction-tuned variants on huggingface and new ones are coming out every day.
I personally had the best experience with GPT4-x-vicuna: https://huggingface.co/NousResearch/gpt4-x-vicuna-13b https://huggingface.co/NousResearch/gpt4-x-vicuna-13b.
There's more variants, and you can find information on them on https://www.reddit.com/r/LocalLLaMA https://www.reddit.com/r/LocalLLaMA.
The best way for you to run the model is probably through https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp. This is a plain C/C++ implementation that can run LLMs pretty efficiently. It can run the LLaMA 13B variants at a pretty quick pace (~100 ms/token) on my M1 Pro macbook.
I'd be happy to answer more questions.
- behnamoh 3y agoThank you. As I'm new to this field, I have been grappling with some concepts like GGML vs GPTQ models, etc. It seems the GPTQ models only run on Linux, so your suggestion is right to use llama.cpp as that works with GGML and is compatible with M1 Macs as well. I had trouble running oobabooga on M1 Mac, something about Python arm64 vs x86_64... I didn't know about GPT4-x-Vicuna that you mentioned. Interesting to see that it's uncensored. I knew about Wizard-Vicuna-uncensored though. It seems like GPT4-x-Vicuna is similar.