3 ms·

I just used their `Bonsai-demo` repo like this; `cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh` then used it in a very minimalisti

I just used their `Bonsai-demo` repo like this; `cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh` then used it in a very minimalistic pi with a very small system prompt. Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.

yearolinuxdsktp 7d ago
Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
aetherspawn 6d ago
Yes, I use LM Studio with MLX support, which is specifically faster on M series Macs. I am not sure if llama is the same, but I guess what I’m saying is if you want the performance to be good on M series you have to use models packaged in the right format.