3 ms·
Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 F
by alsima 2mo ago
Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 Flash wouldn't preserve quality.
- seizethecheese 2mo agoMentioned where? What do you mean it wouldn't preserve quality? I demonstrate in the comments here how it would on a saturated benchmark.
- alsima 2mo agoMentioned in my comment above. Sorry, I thought you were talking about Gemini 3.6 Flash, not their new one. Yes, that's the reason why we didn't include the router in the benchmark to have a fair comparison of the harness itself.