As a side project I've been writing a tool that drives GPT4 for coding purposes. It finds FIXME comments and equips the LLM with a set of high level shell comma

As a side project I've been writing a tool that drives GPT4 for coding purposes. It finds FIXME comments and equips the LLM with a set of high level shell commands it can use to explore the codebase and edit the files. This works well and I find it superior to the (often semi-broken) chat UI that OpenAI provide. There's no need for copy paste, and it can be given instructions like // FIXME: Implement this class paying attention to the unit tests at which point it will go look up the unit test class, read it, understand what the code needs to do and proceed to edit the necessary algorithms into the code. The tool is a mix of response parsing/execution and prompt building, so it can do builds/tests/fix cycles. At the end you get a git branch with the work it's done. By this point I'm kinda developing an intuition for what GPT4 can and can't do, but whilst pushing against what it can do I still often find myself being impressed. Maybe not "mind blown" anymore because it's amazing how quickly you can get used to this stuff, but still. Not only does it do an excellent job of figuring out what to do when the instructions are clear enough, but it also has commands for adding library dependencies and will use relevant open source libs to make its job easier. The hard part about using this sort of tool is that you can quickly become bottlenecked on figuring out what it is you actually want. It's easy to run ahead of yourself and end up with a mess. This sort of very rapid "painting with code" feel is a bit new; probably the slowness of typing allows your subconscious to think ahead a bit when doing programming normally. Also, I'm still teaching it how to explore the codebase efficiently so things that require a lot of context aren't quite there yet. I've got a whole plan for how to make that work better though. I expect at some point soon some big company or some $100M "seed round" startup will come along and do this better, but I didn't want to wait.