After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests. A simple, naive solution from these LLM

After the recent spate of announcements for coding LLMs, I dug up a problem from my early days in programming contests. A simple, naive solution from these LLMs runs 1000-2000x slower than the winning solution depending on the algorithm the LLM uses. Asking the LLM to "make it faster" can knock the runtime by 10x. Close but no cigar. Further prodding for even faster solutions sometimes shows glimmers of bridging the mental gap - but inevitably, the LLM doesn't know what to do with the newfound knowledge and wallows in its local minimum. Typically ending with "this is the fastest code to solve the problem". Oh, grasshopper. You fail. It wouldn't be so bad if this problem solving session was consistently reproducible. But every time I create a new session and give the problem, often times the naive solution appears. Sometimes the LLM can't even produce a working naive solution - even though the comments in the proposed source code shows the LLM knows what the correct answer is - the actual source code can't produce that answer. Once, I decided to give some hints to the LLM to see if I could get it to jump the chasm. The LLM decided to limit its search space to a certain range - which I knew was wrong, and resulted in incorrect results. I prompted the LLM to try and expand the search range, which it cheerfully tried and deduced an expanded search range - which was still too limited. Further prodding resulted in the LLM reverting back to the original search range. No amount of prodding would get the LLM to expand its search range. In a fit of rage, I told the LLM to use an arbitrary large search range which would cover the necessary search space. The LLM refused! Dang - I was chatting with HAL from 2001 - A Space Odyssey. I have been unable to repro that particular chat situation. I guess we should be happy that the LLM can even produce a working solution - probably 5-10 years ago, we'd be gobsmacked. Even call it magic... But it's not remotely consistent.