4 ms·

When reading this type of announcements, always have keen eyes on graphs. e.g. "Agent Coding Performance by Effort Level" cuts Y-axis from 0~20. - This makes

When reading this type of announcements, always have keen eyes on graphs. e.g. "Agent Coding Performance by Effort Level" cuts Y-axis from 0~20. - This makes it as if GLM-5.3-Flash made a bigger jump than it claimed as the Y-axis does not increase much (stupid trick used in biz reports) I did mention that ox was working ok for me, and having an open-weight comparable to close to SOTA makes it very compelling for me to try it out locally (well, only if I got more VRAM)

nchmy 1mo ago
they also conspicuously omitted GPT 5.6 Luna from comparison. It scores lower, but is also cheaper. MiMo 2.5 is not a valid comp at this point edit: nevermind. it is there in the artifical analysis scatter plot, but is greyed-out. MUCH more interesting is that in that chart, their cost is WAY off. The actual chart shows GLM 5.3 Flash at $0.09, but their chart shows $0.045...
- mrtesthah 1mo ago
  The web page says 5.3 flash is discounted right now.
  - drob518 1mo ago
    Seems disingenuous to draw frontier graphs with starter pricing.
    - seaal 1mo ago
      Well, Luna debuted with 5x higher pricing than is currently available. With the pace of recent development these models might not be relevant by Thanksgiving.
      - drob518 1mo ago
        Of course. Pricing is always changing, but typically it goes down over time, not up. So, if you're showing artificially low pricing from the start based on a teaser rate, IMO, you shouldn't be using that to show where you appear on a frontier graph. Place yourself on the graph based on your expected long-term pricing. Then, over time, adjust your position based on your standard rate, whatever that might be. Games are always being played for things like this, but this seems excessive.
        yipinwong 1mo ago
        I don't know if that's the standard pricing for US models to go down overtime, while Chinese ones go up (start cheap but pay more). I don't have enough metrics to compare those costs but still Chinese models have been cheaper except against Luna for me. FWIW, Luna does everything so well, I just keep using it for all my agents by default.
        drob518 1mo ago
        I haven't noticed the Chinese models going up in price for the same model. They do release new versions of the models with different prices that are higher. But everybody is doing that. One fine point is that deepseek-v4-flash-0731 is really a different model than deepseek-v4-flash and it's priced higher.

nchmy 1mo ago

they also conspicuously omitted GPT 5.6 Luna from comparison. It scores lower, but is also cheaper. MiMo 2.5 is not a valid comp at this point edit: nevermind. it is there in the artifical analysis scatter plot, but is greyed-out. MUCH more interesting is that in that chart, their cost is WAY off. The actual chart shows GLM 5.3 Flash at $0.09, but their chart shows $0.045...

mrtesthah 1mo ago
The web page says 5.3 flash is discounted right now.
- drob518 1mo ago
  Seems disingenuous to draw frontier graphs with starter pricing.
  - seaal 1mo ago
    Well, Luna debuted with 5x higher pricing than is currently available. With the pace of recent development these models might not be relevant by Thanksgiving.
    - drob518 1mo ago
      Of course. Pricing is always changing, but typically it goes down over time, not up. So, if you're showing artificially low pricing from the start based on a teaser rate, IMO, you shouldn't be using that to show where you appear on a frontier graph. Place yourself on the graph based on your expected long-term pricing. Then, over time, adjust your position based on your standard rate, whatever that might be. Games are always being played for things like this, but this seems excessive.
      - yipinwong 1mo ago
        I don't know if that's the standard pricing for US models to go down overtime, while Chinese ones go up (start cheap but pay more). I don't have enough metrics to compare those costs but still Chinese models have been cheaper except against Luna for me. FWIW, Luna does everything so well, I just keep using it for all my agents by default.
        drob518 1mo ago
        I haven't noticed the Chinese models going up in price for the same model. They do release new versions of the models with different prices that are higher. But everybody is doing that. One fine point is that deepseek-v4-flash-0731 is really a different model than deepseek-v4-flash and it's priced higher.