3 ms·
I'm a bit curious to see how close the solutions were. When it couldn't solve it within the constraints, was it 'broadly correct' with some bugs? Was it on the
by xen0 2y ago
I'm a bit curious to see how close the solutions were.
When it couldn't solve it within the constraints, was it 'broadly correct' with some bugs? Was it on the right track or completely off?
- jerpint 2y agoThe code they generated and the outputs (including trace back errors) is all included, you can view them on the post itself or from hf space: https://huggingface.co/spaces/jerpint/advent24-llm https://huggingface.co/spaces/jerpint/advent24-llm
- xen0 2y agoHuh, don't know how I missed that. I was hoping for them to have done some analysis themselves. Glancing through the code for some of the 'easier' problems Claude (his best performer), it seems to have broadly correct (if a little strange and overwrought) code. But my phone is not a good platform for me to do this kind of reading.
- jerpint 2y agoI should specify that unfortunately I didn’t store the raw output from the LLM, just the parsed code snippets they produced, but the code to reproduce it is all there