5 ms·

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the

> 225 coding exercises from Exercism Has there been any effort taken to reduce data leakage of this test set? Sounds like these exercises were available on the internet pre-2023, so they'll probably be included in the training data for any modern model, no?

anotherpaulg 2y ago
I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months. To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.
- szundi 2y ago
  [dead]
- jrflowers 2y ago
  >I try not to let perfect be the enemy of good. All benchmarks have limitations. Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard
- Marazan 2y ago
  Having the verbatim answer to the test is not a "limitation" it is an invalidation.
  - deleted 2y ago
    [deleted]
- rodrigodlu 2y ago
  That's my perception as well. Most of the time, most of the devs I know, including myself, are not really creating novelty with the code itself, but with the product. (Sometimes even the product is not novel, just a similar enhanced version of existing products) If the resulting code is not trying to be excessively clever or creative this is actually a good thing in my book. The novelty and creativity should come from the product itself, especially from the users/customers perspective. Some people are too attached to LLM leaderboards being about novelty. I want reliable results whenever I give the instructions, either be the code, or the specs built into a spec file after throwing some ideas into prompts.
- guccihat 2y ago
  > The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code The Aider Polyglot website also states that the benchmark " ...asks the LLM to edit source files to complete 225 coding exercises". However, when looking at the actual tests [0], it is not about editing code bases, it's rather just solving simple programming exercies? What am I missing? [0] https://github.com/Aider-AI/polyglot-benchmark https://github.com/Aider-AI/polyglot-benchmark
chvid 2y ago
They leak the second they are used on a model behind an API, don't they?
- chvid 2y ago
  As far as I can tell the only way of doing a comparison of two models, that cannot be easily gamed, is being having them in open weights form and then running them against a benchmark that was created after both of the two models were created.
jonplackett 2y ago
I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temperature and why? Phrased this way without any help, all but the thinking models get it wrong
- akoboldfrying 2y ago
  The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too. I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.
  - xx_ns 2y ago
    Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one. EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?
    - danbruc 2y ago
      That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.

anotherpaulg 2y ago

I try not to let perfect be the enemy of good. All benchmarks have limitations. The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months. To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.

szundi 2y ago
[dead]
jrflowers 2y ago
>I try not to let perfect be the enemy of good. All benchmarks have limitations. Overfitting is one of the fundamental issues to contend with when trying to figure out if any type of model at all is useful. If your leaderboard corresponds to vibes and that is your target, you could just have a vibes leaderboard
Marazan 2y ago
Having the verbatim answer to the test is not a "limitation" it is an invalidation.
- deleted 2y ago
  [deleted]
rodrigodlu 2y ago
That's my perception as well. Most of the time, most of the devs I know, including myself, are not really creating novelty with the code itself, but with the product. (Sometimes even the product is not novel, just a similar enhanced version of existing products) If the resulting code is not trying to be excessively clever or creative this is actually a good thing in my book. The novelty and creativity should come from the product itself, especially from the users/customers perspective. Some people are too attached to LLM leaderboards being about novelty. I want reliable results whenever I give the instructions, either be the code, or the specs built into a spec file after throwing some ideas into prompts.
guccihat 2y ago
> The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code The Aider Polyglot website also states that the benchmark " ...asks the LLM to edit source files to complete 225 coding exercises". However, when looking at the actual tests [0], it is not about editing code bases, it's rather just solving simple programming exercies? What am I missing? [0] https://github.com/Aider-AI/polyglot-benchmark https://github.com/Aider-AI/polyglot-benchmark

chvid 2y ago

They leak the second they are used on a model behind an API, don't they?

chvid 2y ago
As far as I can tell the only way of doing a comparison of two models, that cannot be easily gamed, is being having them in open weights form and then running them against a benchmark that was created after both of the two models were created.

jonplackett 2y ago

I like to make up my own tests, that way you know it is actually thinking. Tests that require thinking about the physical world are the most revealing. My new favourite is: You have 2 minutes to cool down a cup of coffee to the lowest temp you can. You have two options: 1. Add cold milk immediately, then let it sit for 2 mins. 2. Let it sit for 2 mins, then add cold milk. Which one cools the coffee to the lowest temperature and why? Phrased this way without any help, all but the thinking models get it wrong

akoboldfrying 2y ago
The fact that the answer is interesting makes me suspect that it's not a good test for thinking. I remember reading the explanation for the answer somewhere on the internet years ago, and it's stayed with me ever since. It's interesting enough that it's probably been written about multiple times in multiple places. So I think it would probably stay with a transformer trained on large volumes of data from the internet too. I think a better test of thinking is to provide detail about something so mundane and esoteric that no one would have ever thought to communicate it to other people for entertainment, and then ask it a question about that pile of boring details.
- xx_ns 2y ago
  Out of curiosity, what is the answer? From your comment, it seems like the more obvious choice is the incorrect one. EDIT: By the more obvious one, I mean letting it cool and then adding milk. As the temperature difference between the coffee and the surrounding air is higher, the coffee cools down faster. Is this wrong?
  - danbruc 2y ago
    That is the correct answer. Also there is a lot of potential nuance, like evaporation or when you take the milk out of the fridge or the specific temperatures of everything, but under realistic settings adding the milk late will get you the colder coffee.