3 ms·

Gemini 3.8 flash seems to (subjectively) be the outlier in terms of performance to cost ratio?

• GaggiX 12d ago
  Gemini 3.8 Flash results are often not very coherent but it does put a lot of shading and details to hide the fact.
• pixelesque 12d ago
  Its giraffe / grandfather clock one is pretty bad... (two necks? wearing a suit?) Weird as well, it's clearly pulled out some 1884 patent on clock designs, and a quick ddg/google doesn't show it as anything to do with grandfather clocks.
• simonw 12d ago
  Google Gemini are the only team who have openly had staff deliberately spend time on SVG performance: https://twitter.com/sunjiao123sun_/status/2024555516551372920 https://twitter.com/sunjiao123sun_/status/202455551655137292... See also this Jeff Dean tweet showing off their animals-in-vehicles abilities: https://twitter.com/JeffDean/status/2024525132266688757 https://twitter.com/JeffDean/status/2024525132266688757
  • gpt5 12d ago
    It’s very likely they all add svg generation into the training data. It’s part of the reason it’s no longer a good benchmark (unless you need to generate SVGs).
    • simonw 12d ago
      It's about more than just training data. Every model since GPT-2 had SVGs in the training data, because they all used a scrape of the Web and the Web is full of SVGs. What's unique about Gemini is that they actively worked to get better results for their SVGs - maybe RLHF, maybe RLVF of some sort.