> You need to already know what you're doing to know whether the LLM got it right. I want to call out that this is not necessarily true. You can interact with

> You need to already know what you're doing to know whether the LLM got it right. I want to call out that this is not necessarily true. You can interact with the LLM using agents, feedback loops, and some scoring function on randomized test inputs to produce novel code that you don't know the structure of beforehand. You start with a set of example inputs and desired outputs, and prompt the llm for a function in your programming language of choice. You then take the LLM's response and feed it into an agent that executes the code against the inputs and outputs, and report the discrepancies back to the LLM. It responds with a new completion, which you feed to the agent until all your example inputs produce the expected outputs. Finally, you can use property-based testing to produce new examples for testing by the llm, resolving the answers until the produced code is correct to within some margin of acceptable correctness. You can do this to produce code without needing to know anything besides the desired properties of generated code and the properties of the inputs. You can further automate this by using a separate LLM to produce the examples instead of the typical generation and shrinking functions to produce examples. This doesn't require any prior knowledge of the target language. You can expand beyond programming into any domain for which you can produce a scoring function and automate input generation. I suspect that you can use control theory (PID loops, behavior choice loops scored by Eigenvalues) to model complex scenarios spanning multiple domains as well, choosing the evaluation agent as the behavior, each behavior of which has a separately defined scoring function. All of this can be automated by using the LLM generator without prior knowledge of the algorithmic structure of the solution and knowledge of all possible inputs to all possible behaviors. If that all sounds very familiar, it's because it's essentially just doing test-driven development, but with the LLM machine as the developer. It would be expensive to run an entire project development that way, due to the high costs of executing LLMs and/or querying LLM apis, but if the cost is less than that of employing a junior developer or teams of developers, it might be worth it. And human intervention can be kept in the loop at any stage, making it very feasible for rapid prototyping - where you don't necessarily need the entirely correct answer from the machine, just a good enough starting point to allow the human to take over to produce a result.