3 ms·

It's serviceable but, like many Chinese models, it uses a lot of tokens to get work done.

by esafak 3mo ago

gruez 3mo ago
>it uses a lot of tokens to get work done. That's irrelevant when you use $/task as the metric, which the OP does use.
- esafak 3mo ago
  It also affects the time.
aitchnyu 3mo ago
It felt like a rocket compared to GLM 5.2 though. Are Chinese models generally token-heavy?
- esafak 3mo ago
  https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-task&models=gpt-5-6-terra-medium%2Cgpt-5-6-luna-medium%2Cgpt-5-6-terra-high%2Cgpt-5-6-luna-xhigh%2Cgpt-5-6-terra-xhigh%2Cgpt-5-6-luna-high%2Cglm-5-2%2Cqwen3-8-max%2Cdeepseek-v4-flash&intelligence-category=reasoning-vs-non-reasoning&intelligence-efficiency=output-tokens-per-task#intelligence-efficiency-tabs https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...
cyanydeez 3mo ago
If I had the GPU size, hook it up to llama.cpp and setup the --reasoning-budget and reasoning-message; Most of that additional reasoning is a lot of garbage and you can redirect it to useful output. That's how I handle the Qwen27B and 35B
- nomel 3mo ago
  > Most of that additional reasoning is a lot of garbage and you can redirect it to useful output. What do you mean by "redirect it to useful output"? Could you give an example? This sounds interesting.
  - cyanydeez 3mo ago
    It's specific to the harness. Using dynamic context pruning, the budget cuts it off after a select amount of tokens and the budget message tells the model to use subgents to finish whatever it's thinking about
    - nomel 3mo ago
      Nice. Does it use a summarization, or a hard cutoff?
      - ttkciar 3mo ago
        llama.cpp uses a hard cutoff. The agent then does "something" that is specific to the agent's implementation and configuration. It might summarize and then "finish the thought" with a different model, and then resubmit the prompt to the llama.cpp API endpoint with <think>..</think> prefilled. The primary model then infers the remainder of the reply.
        cyanydeez 3mo ago
        llama.cpp does a hard cut off on budget; it can set a reasoning-message as default but the client _can_ set a per message reasoning-message, so it's possible a smart harness could inspect the cut of thoughts and trim and do whatever.

It's serviceable but, like many Chinese models, it uses a lot of tokens to get work done.

by esafak 3mo ago

It's serviceable but, like many Chinese models, it uses a lot of tokens to get work done.

gruez 3mo ago

>it uses a lot of tokens to get work done. That's irrelevant when you use $/task as the metric, which the OP does use.

esafak 3mo ago
It also affects the time.

aitchnyu 3mo ago

It felt like a rocket compared to GLM 5.2 though. Are Chinese models generally token-heavy?

esafak 3mo ago
https://artificialanalysis.ai/?cost=intelligence-vs-cost-per-task&models=gpt-5-6-terra-medium%2Cgpt-5-6-luna-medium%2Cgpt-5-6-terra-high%2Cgpt-5-6-luna-xhigh%2Cgpt-5-6-terra-xhigh%2Cgpt-5-6-luna-high%2Cglm-5-2%2Cqwen3-8-max%2Cdeepseek-v4-flash&intelligence-category=reasoning-vs-non-reasoning&intelligence-efficiency=output-tokens-per-task#intelligence-efficiency-tabs https://artificialanalysis.ai/?cost=intelligence-vs-cost-per...

cyanydeez 3mo ago

If I had the GPU size, hook it up to llama.cpp and setup the --reasoning-budget and reasoning-message; Most of that additional reasoning is a lot of garbage and you can redirect it to useful output. That's how I handle the Qwen27B and 35B

nomel 3mo ago
> Most of that additional reasoning is a lot of garbage and you can redirect it to useful output. What do you mean by "redirect it to useful output"? Could you give an example? This sounds interesting.
- cyanydeez 3mo ago
  It's specific to the harness. Using dynamic context pruning, the budget cuts it off after a select amount of tokens and the budget message tells the model to use subgents to finish whatever it's thinking about
  - nomel 3mo ago
    Nice. Does it use a summarization, or a hard cutoff?
    - ttkciar 3mo ago
      llama.cpp uses a hard cutoff. The agent then does "something" that is specific to the agent's implementation and configuration. It might summarize and then "finish the thought" with a different model, and then resubmit the prompt to the llama.cpp API endpoint with <think>..</think> prefilled. The primary model then infers the remainder of the reply.
      - cyanydeez 3mo ago
        llama.cpp does a hard cut off on budget; it can set a reasoning-message as default but the client _can_ set a per message reasoning-message, so it's possible a smart harness could inspect the cut of thoughts and trim and do whatever.