5 ms·
If you ran the same query set 30x or 15x on the cheaper model (and compensated for all the extra tokens the reasoning model uses), would you be able to realize
by alwa 2y ago
If you ran the same query set 30x or 15x on the cheaper model (and compensated for all the extra tokens the reasoning model uses), would you be able to realize the same 26% quality gain in a machine-adjudicatible kind of way?
- j_maffe 2y agowith a reasoning model you'd get better than both.
- MattSayar 2y agoExactly. Not sure why you'd pick GPT 4.5 over lots of GPT 4o queries or an o1 query
- smohare 2y agoIgnoring latency for a second, one of the tricks for boosting quality is to utilize consensus. One probability does not need to call the lesser model 30x as much to achieve these gains sorta of gains. Moreover you have to take the purported gains with a grain of salt. The models are probably trained on the evaluation sets they are benchmarked against.