4 ms·
Those were still quantized right? My understanding is that LLaMA3 does not quantize well at all.
by anxman 2y ago
Those were still quantized right? My understanding is that LLaMA3 does not quantize well at all.
- h_tbob 2y agoI run llama 3 8B q 4 k m locally, and it's amazing! This is my system prompt: You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.