4 ms·

Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off II

by ein0p 1y ago

Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).

gundmc 1y ago
It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.
rfw300 1y ago
o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.
- ein0p 1y ago
  These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
  - overfeed 1y ago
    > These small models only make sense with "thinking" enabled This entirely depends on your use-cases.
vessenes 1y ago
o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.
- mccraveiro 1y ago
  2.5 flash with reasoning is just 20% cheaper than o4-mini
  - vessenes 1y ago
    Good point: reasoning costs more. Also impossible to tell without tests is how verbose the reasoning mode is
mupuff1234 1y ago
Not sure "decimated" is a fitting word for "slightly higher performance on some benchmarks".
- fwip 1y ago
  Perhaps they were using the original meaning of "one-tenth destroyed." :P
- ein0p 1y ago
  66.8% error rate reduction for o4-mini on AIME2025, and 21% error rate reduction on MMMU isn't "slightly higher". It'll be quite noticeable in practice.
  - deleted 1y ago
    [deleted]
kfajdsl 1y ago
Anecdotally o4-mini doesn’t perform as well on video understanding tasks in our pipeline, and also in Cursor it seems really not great. During one session, it read the same file (same lines) several times, ran ‘python -c ‘print(“skip!”)’’ for no reason, and then got into another file reading loop. Then after asking a hypothetical about the potential performance implications of different ffmpeg flags, it claimed that it ran a test and determined conclusively that one particular set was faster, even though it hadn’t even attempted a tool call, let alone have the results from a test that didn’t exist.

by ein0p 1y ago

Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).

gundmc 1y ago

It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.

rfw300 1y ago

o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.

ein0p 1y ago
These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.
- overfeed 1y ago
  > These small models only make sense with "thinking" enabled This entirely depends on your use-cases.

vessenes 1y ago

o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.

mccraveiro 1y ago
2.5 flash with reasoning is just 20% cheaper than o4-mini
- vessenes 1y ago
  Good point: reasoning costs more. Also impossible to tell without tests is how verbose the reasoning mode is

mupuff1234 1y ago

Not sure "decimated" is a fitting word for "slightly higher performance on some benchmarks".

fwip 1y ago
Perhaps they were using the original meaning of "one-tenth destroyed." :P
ein0p 1y ago
66.8% error rate reduction for o4-mini on AIME2025, and 21% error rate reduction on MMMU isn't "slightly higher". It'll be quite noticeable in practice.
- deleted 1y ago
  [deleted]

kfajdsl 1y ago

Anecdotally o4-mini doesn’t perform as well on video understanding tasks in our pipeline, and also in Cursor it seems really not great. During one session, it read the same file (same lines) several times, ran ‘python -c ‘print(“skip!”)’’ for no reason, and then got into another file reading loop. Then after asking a hypothetical about the potential performance implications of different ffmpeg flags, it claimed that it ran a test and determined conclusively that one particular set was faster, even though it hadn’t even attempted a tool call, let alone have the results from a test that didn’t exist.