parched99

16 ms·

1. 1.▲
   by parched99 1y ago

   I answered the question directly. IQ4_X_S is smaller, but slower and less accurate than Q4_0. The parent comment specifically asked about the QAT version. That's literally what this thread is about. The context-length mention was relev
2. 2.▲
   by parched99 1y ago

   Resolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked. Best to have two or more low-end, 16GB GPUs for a total of 32GB VRA
3. 3.▲
   by parched99 1y ago

   I'm aware. I was addressing the question being asked.
4. 4.▲
   by parched99 1y ago

   I think Powershell is a bad test. I've noticed all local models have trouble providing accurate responses to Powershell-related prompts. Strangely, even Microsoft's model, Phi 4, is bad at answering these questions without careful
5. 5.▲
   by parched99 1y ago

   I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 4