3 ms·

In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough

In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough better to justify moving away from 3.6 35B even though 35B is still faster. The Unsloth weights don't come with vision support but you can add that back yourself. llama.cpp recipe in case anyone else wants to save some time on setup: https://pastebin.com/fcqsbDTv https://pastebin.com/fcqsbDTv

mdp2021 2mo ago
> Better... than 3.8 27B How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation. Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...
- hedgehog 2mo ago
  Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.
  - mdp2021 2mo ago
    Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.
    - mapontosevenths 2mo ago
      MOEs are built by training a second "router" model to identify which parts matter inside the dense model. Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes. Modern LLM's are wildly inefficient.
data-ottawa 2mo ago
We should still expect significant performance improvements. I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
- hedgehog 2mo ago
  The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.
  - cmrdporcupine 2mo ago
    My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload. It's not world changing, but at those speeds I'll take anything I can get. (The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing). https://github.com/rdaum/eider/ https://github.com/rdaum/eider/
    - hedgehog 2mo ago
      It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.
      - cmrdporcupine 2mo ago
        Probably. I've spent zero time with optimization at this point. Code is all new this morning. Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk. Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
        2mo ago

mdp2021 2mo ago

> Better... than 3.8 27B How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation. Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...

hedgehog 2mo ago
Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.
- mdp2021 2mo ago
  Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.
  - mapontosevenths 2mo ago
    MOEs are built by training a second "router" model to identify which parts matter inside the dense model. Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes. Modern LLM's are wildly inefficient.

data-ottawa 2mo ago

We should still expect significant performance improvements. I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.

hedgehog 2mo ago
The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.
- cmrdporcupine 2mo ago
  My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload. It's not world changing, but at those speeds I'll take anything I can get. (The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing). https://github.com/rdaum/eider/ https://github.com/rdaum/eider/
  - hedgehog 2mo ago
    It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.
    - cmrdporcupine 2mo ago
      Probably. I've spent zero time with optimization at this point. Code is all new this morning. Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk. Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?
      - 2mo ago