If you give it a reasonably well specified and well understood problem, solvable in a page or two of code and let it run in a loop (giving feedback) it will wri

If you give it a reasonably well specified and well understood problem, solvable in a page or two of code and let it run in a loop (giving feedback) it will write and debug the code by itself. The output might not be "great" but it tends to be serviceable. The problems with copy/pasting from a ChatGPT window are often environmental; modern computing environments are diverse and not well suited to one-shotting problems. I find it quickly corrects its own hallucinations under feedback and can understand what it needs to fill in. My experience has been that the "bullshit" problems go away when there's a concrete problem and reliable feedback. This is not hard to automate and I imagine we will see TDD-based LLM "programming loops" where the human specifies problem, LLM generates tests, human supervises / extends tests, then LLM generates code to solve problem in the next 12mo. I feel like I can guarantee there are people working on frameworks for this right now. Bear in mind that we are very early on the whole journey and that LLMs are likely to be only one part of a solution that can solve bigger problems. The obvious problems using an LLM naively are the "context window" and deciding on appropriate sub-problems. Being able to create good working memory seems like a priority, whether that is vector databases or some other thing. I don't think it will be able to work in "novel" domains for some time and will not generally compete with skilled programmers yet. But with the right architecture invoking it I'm highly confident that even today's LLMs will be able to bash out crud apps up to say 50kloc from specifications alone, jankily this year and reliably within 3 years. It's hard to predict just how good it will get and how fast, without understanding what further fundamental advances are possible. [Note I'm just describing solution domains where "traditional" code is the target. I think this is where a lot of programmers suffer from a "skill curse". They think code is needed to solve problems and forget how a McDonald's can profitably emit burgers using nondeterministic teenagers, even if they occasionally forget your fries. It may be the case that "code" itself turns out to be a dead end or "expensive optimisation" for very many business problems. For example, a lot of business runs on forms and they are a staple of in-house applications. For many form processing applications it may make more sense for an LLM to just process a form or request directly and produce side effects and outputs based purely on instructions from the business without bothering with the nuisance of code at all. It will be interesting to see which direction gets more love - LLM just solves problem instances at runtime or LLM doing work at compile time for cost reduction and determinism.]