4 ms·

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.html

doctoboggan 3y ago
Its unclear to me if the 1106 is the same as the turbo model.
- anotherpaulg 3y ago
  Ya, OpenAI's naming schemes don't seem very consistent. But my read of the announcement is that gpt-4-1106-preview is the new turbo model. GPT-4 Turbo is available for all paying developers to try by passing gpt-4-1106-preview in the API and we plan to release the stable production-ready model in the coming weeks. https://openai.com/blog/new-models-and-developer-products-announced-at-devday https://openai.com/blog/new-models-and-developer-products-an...
  - doctoboggan 3y ago
    Ok great thank you for pointing that out.
usaar333 3y ago
I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.
- anotherpaulg 3y ago
  One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4. GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format. GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace. All of this is described and quantified in more detail in the original aider benchmarking writeup: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole. Aider will let you do gpt-4/whole if you'd like to spend the time and money: aider --model gpt-4 --edit-format whole Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.html

doctoboggan 3y ago

Its unclear to me if the 1106 is the same as the turbo model.

anotherpaulg 3y ago
Ya, OpenAI's naming schemes don't seem very consistent. But my read of the announcement is that gpt-4-1106-preview is the new turbo model. GPT-4 Turbo is available for all paying developers to try by passing gpt-4-1106-preview in the API and we plan to release the stable production-ready model in the coming weeks. https://openai.com/blog/new-models-and-developer-products-announced-at-devday https://openai.com/blog/new-models-and-developer-products-an...
- doctoboggan 3y ago
  Ok great thank you for pointing that out.

usaar333 3y ago

I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.

anotherpaulg 3y ago
One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4. GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format. GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace. All of this is described and quantified in more detail in the original aider benchmarking writeup: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole. Aider will let you do gpt-4/whole if you'd like to spend the time and money: aider --model gpt-4 --edit-format whole Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.