3 ms·
I’ve personally run the Llama 3 models locally on a Mac Studio with 128GB of ram, and found it (and other open source models) to perform highly erratically loca
by wcallahan 2y ago
I’ve personally run the Llama 3 models locally on a Mac Studio with 128GB of ram, and found it (and other open source models) to perform highly erratically locally.
And while I’ve also encountered quality control issues with the open source models in “professionally hosted” settings as well (e.g., Groq, OpenRouter), it was much worse locally.
Sentences ending abruptly and non-sensical/unrelated words were a common problem.
- anxman 2y agoThose were still quantized right? My understanding is that LLaMA3 does not quantize well at all.
- h_tbob 2y agoI run llama 3 8B q 4 k m locally, and it's amazing! This is my system prompt: You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.