4 ms·
My gut feeling is that the softcap for coding skills of current LLMs is somewhere here. There is only so far you can go if your model is a directed graph that c
by chmod775 3y ago
My gut feeling is that the softcap for coding skills of current LLMs is somewhere here. There is only so far you can go if your model is a directed graph that cannot think/symbolically execute while it has to write code by emitting it incrementally top to bottom. A different architecture is needed to go further.
- simonw 3y agoWe already know the trick to get massively improved coding performance from an existing model: set up a mechanism for it to execute the code that it writes and then react to any errors or test failures that occur. ChatGPT Code Interpreter has been demonstrating the effectiveness of this trick for over a year now. Anthropic are working on their own version. I got Code Interpreter to write, compile, debug and fix C code the other day, despite it pretending it can only work with Python: https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
- febed 3y agoDid you have any success in trying automate the whole development/debug loop similar what the auto coder tools are trying to achieve?
- unraveller 3y agoCode interpreter seem to have found a path to perfection, i don't care how bad the first response is (if there have to be >1 turns) as long as we can sync up quickly from any misunderstanding, mine or theirs. Here they kind of made the most easily nudgeable code LLM and won the benchmarks that way. https://opencodeinterpreter.github.io https://opencodeinterpreter.github.io