I posted in a similar thread yesterday. I recently completed a fairly extensive benchmarking effort, where I asked GPT-3.5 and GPT-4 to solve 133 exercises from

I posted in a similar thread yesterday. I recently completed a fairly extensive benchmarking effort, where I asked GPT-3.5 and GPT-4 to solve 133 exercises from the Exercism Python practice set. I measured the performance of the Feb (0301) and June (0613) versions of gpt-3.5-turbo and gpt-4. This is the only systematic, quantitative evaluation that I am aware of that compares the different versions of the OpenAI models over time. My conclusions: - The new June GPT-3.5 models did a bit worse than the old Feb model. - For GPT-4, there wasn't much difference between June and Feb. June was maybe a bit better. - It hurts coding performance to have GPT package up code inside the new function calls API. - As expected, GPT-4 is better than GPT-3.5 at code editing. All the details are written up here: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html Some specific notes about GPT-3.5 getting a bit worse in June are here: https://aider.chat/docs/benchmarks.html#the-0613-models-seem-worse https://aider.chat/docs/benchmarks.html#the-0613-models-seem...