I like the idea, but I feel like the execution left a bit to be desired. My gut tells me you can get much better results from the models with better prompting.

I like the idea, but I feel like the execution left a bit to be desired. My gut tells me you can get much better results from the models with better prompting. The whole "You are solving the 2024 advent of code challenge." form of prompting is just adding noise with no real value. Based on my empirical experience, that likely hurts performance instead of helping. The time limit feels arbitrary and adds nothing to the benchmark. I don't understand why you wouldn't include o1 in the list of models. There's just a lot here that doesn't feel very scientific about this analysis...