3 ms·

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloadi

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.

pulse7 3mo ago
Most probably not optimized yet for this model...
dofm 3mo ago
It’s still a dense model so all 27B parameters are active. 35B-A3B activates only 3B at any time.