3 ms·

Are there any benchmarks that track historical performance?

• behnamoh 1y ago
  good question, and I don't know of any, although it's a no brainer that someone should make it. a proxy to that may be the anecdotal evidence of users who report back in a month that model X has gotten dumber (started with gpt-4 and keeps happening, esp. with Anthro and OpenAI models). I haven't heard such anecdotal stories about Gemini, R1, etc.
• SparkyMcUnicorn 1y ago
  Aider has one, but it hasn't been updated in months. People kept claiming models were getting worse, but the results proved that they weren't.
  • esafak 1y ago
    https://aider.chat/docs/leaderboards/by-release-date.html https://aider.chat/docs/leaderboards/by-release-date.html
  • __mharrison__ 1y ago
    Updated yesterday... https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
    • vitaflo 1y ago
      That Deepseek price is always hilarious to see in these charts.
    • SparkyMcUnicorn 1y ago
      That's not the one I'm referring to. See my other comments or your sibling comment.