5 ms·

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT s

by troad 7mo ago

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere 16 gigs of RAM. The recent Qwen 3.5 9B model is pretty good, though it did trip all over itself to avoid telling me what happened on Tiananmen Square in 1989. (But then I tried something called "Qwen3.5-9B-Uncensored-HauhauCS-Aggressive", which veers so hard the other way that it will happily write up a detailed plan for your upcoming invasion of Belgium, so I guess it all balances out?)

whackernews 7mo ago
Oh does llama.cpp use MLX or whatever? I had this question, wonder if you know? A search suggests it doesn’t but I don’t really understand.
- irusensei 7mo ago
  >Oh does llama.cpp use MLX or whatever? No. It runs on MacOS but uses Metal instead of MLX.
  - zozbot234 7mo ago
    ANE-powered inference (at least for prefill, which is a key bottleneck on pre-M5 platforms) is also in the works, per https://github.com/ggml-org/llama.cpp/issues/10453#issuecomment-4148905254 https://github.com/ggml-org/llama.cpp/issues/10453#issuecomm...
  - OkGoDoIt 7mo ago
    Is that better or worse?
    - irusensei 7mo ago
      Depends. MLX is faster because it has better integration with Apple hardware. On the other hand GGUF is a far more popular format so there will be more programs and model variety. So its kinda like having a very specific diet that you swear is better for you but you can only order food from a few restaurants.
      - drob518 7mo ago
        But you can always fall back to GGUF while waiting for the world to build a few more MLX restaurants. Or something like that; the analogy is a bit stretched.
        irusensei 7mo ago
        Yeah I'm terrible with analogies.
- LoganDark 7mo ago
  llama.cpp uses GGML which uses Metal directly.
theshrike79 7mo ago
Qwen3.5 has tool calling, so you can give it a wikipedia tool which it uses to know what happened in Tiananmen Square without issues =)
- girvo 7mo ago
  I'd recommend it too, because the knowledge cutoff of all the open weight Chinese models (M2.7, Qwen3.5, GLM-5 etc) is earlier than you'd think, so giving it web search (I use `ddgr` with a skill) helps a surprising amount
  - theshrike79 7mo ago
    Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.
    - zozbot234 7mo ago
      > Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. That doesn't fix the "you don't know what you don't know" problem which is huge with smaller models. A bigger model with more world knowledge really is a lot smarter in practice, though at a huge cost in efficiency.
      - theshrike79 7mo ago
        That's the key, it just needs to be smart enough to 1) know it doesn't know and 2) "know a guy" as they say =) (call a tool for the exact information) Picking a model that's juuust smart enough to know it doesn't know is the key.
      - spockz 7mo ago
        Ive always wondered where the inflection point lies between on the one hand trying to train the model on all kinds of data such as Wikipedia/encyclopedia, versus in the system prompt pointing to your local versions of those data sources, perhaps even through a search like api/tool. Is there already some research or experimentation done into this area?
WesolyKubeczek 7mo ago
Cool, I always wanted to invade Belgium. Maybe if my plan is good, I could run a successful gofundme?
- troad 7mo ago
  Hey, if Margaret Thatcher's son can give it a go, why not you? Believe in yourself and reach for those dreams. *sparkle emoji*
austinthetaco 7mo ago
Have you played around with any of the Hermes models? they are supposed to be one of the best at non-refusal while keeping sane.
- troad 7mo ago
  Interesting! Unfortunately, the smallest Hermes 4 model I can see is 14B, which would really strain the limits of my little laptop. The only way I might get acceptable performance would be to run it extremely quantised, but then I probably wouldn't see much improvement over the 9B Qwen.

by troad 7mo ago

whackernews 7mo ago

Oh does llama.cpp use MLX or whatever? I had this question, wonder if you know? A search suggests it doesn’t but I don’t really understand.

irusensei 7mo ago
>Oh does llama.cpp use MLX or whatever? No. It runs on MacOS but uses Metal instead of MLX.
- zozbot234 7mo ago
  ANE-powered inference (at least for prefill, which is a key bottleneck on pre-M5 platforms) is also in the works, per https://github.com/ggml-org/llama.cpp/issues/10453#issuecomment-4148905254 https://github.com/ggml-org/llama.cpp/issues/10453#issuecomm...
- OkGoDoIt 7mo ago
  Is that better or worse?
  - irusensei 7mo ago
    Depends. MLX is faster because it has better integration with Apple hardware. On the other hand GGUF is a far more popular format so there will be more programs and model variety. So its kinda like having a very specific diet that you swear is better for you but you can only order food from a few restaurants.
    - drob518 7mo ago
      But you can always fall back to GGUF while waiting for the world to build a few more MLX restaurants. Or something like that; the analogy is a bit stretched.
      - irusensei 7mo ago
        Yeah I'm terrible with analogies.
LoganDark 7mo ago
llama.cpp uses GGML which uses Metal directly.

theshrike79 7mo ago

Qwen3.5 has tool calling, so you can give it a wikipedia tool which it uses to know what happened in Tiananmen Square without issues =)

girvo 7mo ago
I'd recommend it too, because the knowledge cutoff of all the open weight Chinese models (M2.7, Qwen3.5, GLM-5 etc) is earlier than you'd think, so giving it web search (I use `ddgr` with a skill) helps a surprising amount
- theshrike79 7mo ago
  Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.
  - zozbot234 7mo ago
    > Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. That doesn't fix the "you don't know what you don't know" problem which is huge with smaller models. A bigger model with more world knowledge really is a lot smarter in practice, though at a huge cost in efficiency.
    - theshrike79 7mo ago
      That's the key, it just needs to be smart enough to 1) know it doesn't know and 2) "know a guy" as they say =) (call a tool for the exact information) Picking a model that's juuust smart enough to know it doesn't know is the key.
    - spockz 7mo ago
      Ive always wondered where the inflection point lies between on the one hand trying to train the model on all kinds of data such as Wikipedia/encyclopedia, versus in the system prompt pointing to your local versions of those data sources, perhaps even through a search like api/tool. Is there already some research or experimentation done into this area?

WesolyKubeczek 7mo ago

Cool, I always wanted to invade Belgium. Maybe if my plan is good, I could run a successful gofundme?

troad 7mo ago
Hey, if Margaret Thatcher's son can give it a go, why not you? Believe in yourself and reach for those dreams. *sparkle emoji*

austinthetaco 7mo ago

Have you played around with any of the Hermes models? they are supposed to be one of the best at non-refusal while keeping sane.

troad 7mo ago
Interesting! Unfortunately, the smallest Hermes 4 model I can see is 14B, which would really strain the limits of my little laptop. The only way I might get acceptable performance would be to run it extremely quantised, but then I probably wouldn't see much improvement over the 9B Qwen.