3 ms·
Since you did not give the models a chance to test their code and correct any mistakes, I think a more accurate comparison would be if you compared them against
by bryan0 2y ago
Since you did not give the models a chance to test their code and correct any mistakes, I think a more accurate comparison would be if you compared them against you submitting answers without testing (or even running!) your code first
- angarg12 2y agoPeople keep evaluating LLMs on essentially zero-shotting a perfect solution to a coding problem. Once we use tools to easily iterate on code (e.g. generate, compile, test, use outcome to refine prompt) we will turbocharge LLMs coding abilities.
- rhubarbtree 2y agoThis smacks of "moving the goalposts" just as much as the other side is accused of when unimpressed by advances. It's a reasonable test.
- xen0 2y agoI'm a bit curious to see how close the solutions were. When it couldn't solve it within the constraints, was it 'broadly correct' with some bugs? Was it on the right track or completely off?
- jerpint 2y agoThe code they generated and the outputs (including trace back errors) is all included, you can view them on the post itself or from hf space: https://huggingface.co/spaces/jerpint/advent24-llm https://huggingface.co/spaces/jerpint/advent24-llm
- xen0 2y agoHuh, don't know how I missed that. I was hoping for them to have done some analysis themselves. Glancing through the code for some of the 'easier' problems Claude (his best performer), it seems to have broadly correct (if a little strange and overwrought) code. But my phone is not a good platform for me to do this kind of reading.
- jerpint 2y agoI should specify that unfortunately I didn’t store the raw output from the LLM, just the parsed code snippets they produced, but the code to reproduce it is all there