5 ms·

From OpenAI's announcement: > Qodo tested GPT‑4.1 head-to-head against Claude Sonnet 3.7 on generating high-quality code reviews from GitHub pull requests. Acr

From OpenAI's announcement: > Qodo tested GPT‑4.1 head-to-head against Claude Sonnet 3.7 on generating high-quality code reviews from GitHub pull requests. Across 200 real-world pull requests with the same prompts and conditions, they found that GPT‑4.1 produced the better suggestion in 55% of cases. Notably, they found that GPT‑4.1 excels at both precision (knowing when not to make suggestions) and comprehensiveness (providing thorough analysis when warranted). https://www.qodo.ai/blog/benchmarked-gpt-4-1/ https://www.qodo.ai/blog/benchmarked-gpt-4-1/

coldcache 1y ago
Interesting link. Worth noting that the pull requests were judged by o3-mini. Further, I'm not sure that 55% vs 45% is a huge difference.
- marsh_mellow 1y ago
  Good point. They said they validated the results by testing with other models (including Claude), as well as with manual sanity checks. 55% to 45% definitely isn't a blowout but it is meaningful — in terms of ELO it equates to about a 36 point difference. So not in a different league but definitely a clear edge
- elAhmo 1y ago
  I first read it as 55% better, which sounds significantly higher than ~22% which they report here. Sounds misleading.
- joshgachnang 1y ago
  Maybe not as much to us, but for people building these tools, 4.1 being significantly cheaper than Clause 3.7 is a huge difference.
InkCanon 1y ago
>4.1 Was better in 55% of cases Um, isn't that just a fancy way of saying it is slightly better >Score of 6.81 against 6.66 So very slightly better
- kevmo314 1y ago
  A great way to upsell 2% better! I should start doing that.
  - neuroelectron 1y ago
    Good marketing if you're selling a discount all purpose cleaner, not so much for an API.
- marsh_mellow 1y ago
  I don't think the absolute score means much — judge models have a tendency to score around 7/10 lol 55% vs. 45% equates to about a 36 point difference in ELO. in chess that would be two players in the same league but one with a clear edge
  - kevmo314 1y ago
    Rarely are two models put head-to-head though. If Claude Sonnet 3.7 isn't able to generate a good PR review (for whatever reason), a 2% better review isn't all that strong of a value proposition.
    - swyx 1y ago
      the point is oai is saying they have a viable Claude Sonnet competitor now
- wiz21c 1y ago
  "they found that GPT‑4.1 excels at both precision..." They didn't say it is better than Claude at precision etc. Just that it excels. Unfortunately, AI has still not concluded that manipulations by the marketing dept is a plague...
jsnell 1y ago
That's not a lot of samples for such a small effect, I don't think it's statistically significant (p-value of around 10%).
- swyx 1y ago
  is there a shorthand/heuristic to calculate pvalue given n samples and effect size?
  - tedsanders 1y ago
    There are no great shorthands, but here are a few rules of thumb I use: - for N=100, worst case standard error of the mean is ~5% (it shrinks parabolically the further p gets from 50%) - multiply by ~2 to go from standard error of the mean to 95% confidence interval - scale sample size by sqrt(N) So: - N=100: +/- 10% - N=1000: +/- 3% - N=10000: +/- 1% (And if comparing two independent distributions, multiply by sqrt(2). But if they’re measured on the same problems, then instead multiply by between 1 and sqrt(2) to account for them finding the same easy problems easy and hard problems hard - aka positive covariance.)
- marsh_mellow 1y ago
  p-value of 7.9% — so very close to statistical significance. the p-value for GPT-4.1 having a win rate of at least 49% is 4.92%, so we can say conclusively that GPT-4.1 is at least (essentially) evenly matched with Claude Sonnet 3.7, if not better. Given that Claude Sonnet 3.7 has been generally considered to be the best (non-reasoning) model for coding, and given that GPT-4.1 is substantially cheaper ($2/million input, $8/million output vs. $3/million input, $15/million output), I think it's safe to say that this is significant news, although not a game changer
  - jsnell 1y ago
    I make it 8.9% with a binomial test[0]. I rounded that to 10%, because any more precision than that was not justified. Specifically, the results from the blog post are impossible: with 200 samples, you can't possibly have the claimed 54.9/45.1 split of binary outcomes. Either they didn't actually make 200 tests but some other number, they didn't actually get the results they reported, or they did some kind of undocumented data munging like excluding all tied results. In any case, the uncertainty about the input data is larger than the uncertainty from the rounding. [0] In R, binom.test(110, 200, 0.5, alternative="greater")
jacobsenscott 1y ago
That's a marketing page for something called qodo that sells ai code reviews. At no point were the ai code reviews judged by competent engineers. It is just ai generated trash all the way down.

coldcache 1y ago

Interesting link. Worth noting that the pull requests were judged by o3-mini. Further, I'm not sure that 55% vs 45% is a huge difference.

marsh_mellow 1y ago
Good point. They said they validated the results by testing with other models (including Claude), as well as with manual sanity checks. 55% to 45% definitely isn't a blowout but it is meaningful — in terms of ELO it equates to about a 36 point difference. So not in a different league but definitely a clear edge
elAhmo 1y ago
I first read it as 55% better, which sounds significantly higher than ~22% which they report here. Sounds misleading.
joshgachnang 1y ago
Maybe not as much to us, but for people building these tools, 4.1 being significantly cheaper than Clause 3.7 is a huge difference.

InkCanon 1y ago

>4.1 Was better in 55% of cases Um, isn't that just a fancy way of saying it is slightly better >Score of 6.81 against 6.66 So very slightly better

kevmo314 1y ago
A great way to upsell 2% better! I should start doing that.
- neuroelectron 1y ago
  Good marketing if you're selling a discount all purpose cleaner, not so much for an API.
marsh_mellow 1y ago
I don't think the absolute score means much — judge models have a tendency to score around 7/10 lol 55% vs. 45% equates to about a 36 point difference in ELO. in chess that would be two players in the same league but one with a clear edge
- kevmo314 1y ago
  Rarely are two models put head-to-head though. If Claude Sonnet 3.7 isn't able to generate a good PR review (for whatever reason), a 2% better review isn't all that strong of a value proposition.
  - swyx 1y ago
    the point is oai is saying they have a viable Claude Sonnet competitor now
wiz21c 1y ago
"they found that GPT‑4.1 excels at both precision..." They didn't say it is better than Claude at precision etc. Just that it excels. Unfortunately, AI has still not concluded that manipulations by the marketing dept is a plague...

jsnell 1y ago

That's not a lot of samples for such a small effect, I don't think it's statistically significant (p-value of around 10%).

swyx 1y ago
is there a shorthand/heuristic to calculate pvalue given n samples and effect size?
- tedsanders 1y ago
  There are no great shorthands, but here are a few rules of thumb I use: - for N=100, worst case standard error of the mean is ~5% (it shrinks parabolically the further p gets from 50%) - multiply by ~2 to go from standard error of the mean to 95% confidence interval - scale sample size by sqrt(N) So: - N=100: +/- 10% - N=1000: +/- 3% - N=10000: +/- 1% (And if comparing two independent distributions, multiply by sqrt(2). But if they’re measured on the same problems, then instead multiply by between 1 and sqrt(2) to account for them finding the same easy problems easy and hard problems hard - aka positive covariance.)
marsh_mellow 1y ago
p-value of 7.9% — so very close to statistical significance. the p-value for GPT-4.1 having a win rate of at least 49% is 4.92%, so we can say conclusively that GPT-4.1 is at least (essentially) evenly matched with Claude Sonnet 3.7, if not better. Given that Claude Sonnet 3.7 has been generally considered to be the best (non-reasoning) model for coding, and given that GPT-4.1 is substantially cheaper ($2/million input, $8/million output vs. $3/million input, $15/million output), I think it's safe to say that this is significant news, although not a game changer
- jsnell 1y ago
  I make it 8.9% with a binomial test[0]. I rounded that to 10%, because any more precision than that was not justified. Specifically, the results from the blog post are impossible: with 200 samples, you can't possibly have the claimed 54.9/45.1 split of binary outcomes. Either they didn't actually make 200 tests but some other number, they didn't actually get the results they reported, or they did some kind of undocumented data munging like excluding all tied results. In any case, the uncertainty about the input data is larger than the uncertainty from the rounding. [0] In R, binom.test(110, 200, 0.5, alternative="greater")

jacobsenscott 1y ago

That's a marketing page for something called qodo that sells ai code reviews. At no point were the ai code reviews judged by competent engineers. It is just ai generated trash all the way down.