3 ms·

> They do not beat opus on real-world usage We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave

> They do not beat opus on real-world usage We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8. For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC. This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases. As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

cyanydeez 2mo ago
Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.
- spmurrayzzz 2mo ago
  4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
  - croemer 2mo ago
    Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
    - spmurrayzzz 2mo ago
      Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.
croemer 2mo ago
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
- tyre 2mo ago
  0% is not 4% less than 4%, that would be 3.84%. 0% is 4 percentage points (pp) less than 4%.
  - naikrovek 2mo ago
    [dead]
enraged_camel 2mo ago
[flagged]
- spmurrayzzz 2mo ago
  Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
- graceful6800 2mo ago
  They literally stated in their first sentence that it was coding tasks.
HonshinM 2mo ago
[dead]

cyanydeez 2mo ago

Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.

spmurrayzzz 2mo ago
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
- croemer 2mo ago
  Depends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
  - spmurrayzzz 2mo ago
    Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the binomial SE.

croemer 2mo ago

How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.

tyre 2mo ago
0% is not 4% less than 4%, that would be 3.84%. 0% is 4 percentage points (pp) less than 4%.
- naikrovek 2mo ago
  [dead]

enraged_camel 2mo ago

[flagged]

spmurrayzzz 2mo ago
Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
graceful6800 2mo ago
They literally stated in their first sentence that it was coding tasks.

HonshinM 2mo ago

[dead]