5 ms·
OpenAI just released new versions of GPT-3.5 and GPT-4, and there’s a lot of interest about their ability to code compared to the previous versions. With that i
by anotherpaulg 3y ago
OpenAI just released new versions of GPT-3.5 and GPT-4, and there’s a lot of interest about their ability to code compared to the previous versions. With that in mind, I’ve been benchmarking the new models.
These results are preliminiary. OpenAI is enforcing very low rate limits on the new GPT-4 model. I will update the results on this page as quickly my rate limit allows.
- reitzensteinm 3y agoThe legend on the graph lists edit format diff/whole, but I assume that's an error? Which method was used for these tests? Either way, the new model is looking very promising.
- anotherpaulg 3y agoAider uses different prompting strategies for GPT-3.5 and 4. GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format. GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace. All of this is described and quantified in more detail in the original aider benchmarking article. https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html
- reitzensteinm 3y agoMakes sense, thanks. I've so far stuck with "whole" for my own experiments, which is slower and more expensive, but it's a more asynchronous use case compared to Aider.