4 ms·
With no prompt engineering this seems like a weird comparison. I wouldn’t expect anyone to be able to one-shot most of the AOC problems. A fair fight would at l
by yunwal 2y ago
With no prompt engineering this seems like a weird comparison. I wouldn’t expect anyone to be able to one-shot most of the AOC problems. A fair fight would at least use something like cursor’s agent on YOLO mode that can review a command’s output, add logs, etc
- ben_w 2y agoIf you immediately know the candlelight is fire, then the meal was cooked a long time ago. And so it is with success of LLMs in one-shot challenges and any job that depends on such challenges: cooked a long time ago.
- NitpickLawyer 2y agoIndeed. (wild to see a sg-1 quote in the wild!)
- fumeux_fume 2y agoI certainly don't think it's weird to measure one-shot performance on the AOC. Sure, more could be done. More can always be done, but this is still useful and interesting.
- deleted 2y ago[deleted]
- segmondy 2y agoI did zero shot 27 solutions successfully with local model code-qwen2.5-32b. I think adding sonnet or latest gemini will probably get me to 40.
- rhubarbtree 2y agoSeems reasonable to me. More people use ChatGPT than cursor, so use it like ChatGPT. For most coding problems, people won't magically know if the code is "correct" or not, so any one-shot answer that is wrong could be a real hindrance. I don't have time to prompt engineer for every bit of code. I need tools that accelerate my work, not tools that absorb my time.