I mostly try making automated developer scripts, which take a project description as input, and have the LLM write a script to implement it, then execute the sc

I mostly try making automated developer scripts, which take a project description as input, and have the LLM write a script to implement it, then execute the script, then return the output to the LLM, and asked it if the project is complete or it wants to submit an updated version. The sample project description I've been using is asking it to get the public IP, location, weather conditions, and 5 regional news headlines, using free public APIs or feeds. Even with the project description telling it to use free, public, keyless APIs, half the time it will immediately throw in a paid API and a line like weather_api_key = #insert your API key here, the script will run, the API will return a keyerror, we give that to the LLM says "the error is because the API key is missing, let me try again" and submits the exact same code again, when the line about using free APIs is in context. (wttr.in is the public weather API it knows how to use if you remind it specifically that wttr.in exists...) It will also invent indexes when trying to extract the response data. I have tried using 4o, 4o-mini, and mistral small 24b 2501. I get that I still need to do the work of prompt engineering if I want impressive results. I have lots of text strings telling the LLM, you are a developer, you will be given a project description, you will write a script, we will run the script, we will return the output to you, you will judge if the output fully achieves the project description, and if not, include an updated full script to try again. It overall works, but it certainly makes plenty of junior dev mistakes too. We need a way to impart the type of additional knowledge accumulation and instruction following that happens with fine-tuning, at runtime, because clearly, knowledge that is "baked in" and knowledge that is provided in context is in very separate classes.