3 ms·

In the keynote @sama claimed GPT-4-turbo was superior to the older GPT-4. Have any benchmarks or other examples been shown? I am curious to see how much better

In the keynote @sama claimed GPT-4-turbo was superior to the older GPT-4. Have any benchmarks or other examples been shown? I am curious to see how much better it is, if it all. I remember when 3.5 got its turbo version there was some controversy on whether it was really better or not.

somsak2 3y ago
It seems like the "Turbo" models are more about being faster/cheaper, not so much about being better. Kinda similar to the iPhone "S" models or Intel's "tick-tock"
metanonsense 3y ago
It definitely feels worse to me. In the way that GPT3.5 felt worse than GPT4 in the past. Somewhere between 3.5 and 4. Similar to with the Bing plugin activated. Somehow "shallower" and does not seem to grasp my intent as good as before.
anotherpaulg 3y ago
My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.html
- doctoboggan 3y ago
  Its unclear to me if the 1106 is the same as the turbo model.
  - anotherpaulg 3y ago
    Ya, OpenAI's naming schemes don't seem very consistent. But my read of the announcement is that gpt-4-1106-preview is the new turbo model. GPT-4 Turbo is available for all paying developers to try by passing gpt-4-1106-preview in the API and we plan to release the stable production-ready model in the coming weeks. https://openai.com/blog/new-models-and-developer-products-announced-at-devday https://openai.com/blog/new-models-and-developer-products-an...
    - doctoboggan 3y ago
      Ok great thank you for pointing that out.
- usaar333 3y ago
  I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.
  - anotherpaulg 3y ago
    One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4. GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format. GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace. All of this is described and quantified in more detail in the original aider benchmarking writeup: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole. Aider will let you do gpt-4/whole if you'd like to spend the time and money: aider --model gpt-4 --edit-format whole Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.

somsak2 3y ago

It seems like the "Turbo" models are more about being faster/cheaper, not so much about being better. Kinda similar to the iPhone "S" models or Intel's "tick-tock"

metanonsense 3y ago

It definitely feels worse to me. In the way that GPT3.5 felt worse than GPT4 in the past. Somewhere between 3.5 and 4. Similar to with the Bing plugin activated. Somehow "shallower" and does not seem to grasp my intent as good as before.

anotherpaulg 3y ago

My early benchmarking seems to show that it's somewhat better for coding. https://aider.chat/docs/benchmarks-1106.html https://aider.chat/docs/benchmarks-1106.html

doctoboggan 3y ago
Its unclear to me if the 1106 is the same as the turbo model.
- anotherpaulg 3y ago
  Ya, OpenAI's naming schemes don't seem very consistent. But my read of the announcement is that gpt-4-1106-preview is the new turbo model. GPT-4 Turbo is available for all paying developers to try by passing gpt-4-1106-preview in the API and we plan to release the stable production-ready model in the coming weeks. https://openai.com/blog/new-models-and-developer-products-announced-at-devday https://openai.com/blog/new-models-and-developer-products-an...
  - doctoboggan 3y ago
    Ok great thank you for pointing that out.
usaar333 3y ago
I'm really surprised your benchmark shows gpt-3.5-turbo-0301 outperforming gpt-4 (non tubo) on first try coding problems.
- anotherpaulg 3y ago
  One thing to notice is that the gpt-3.5 bars are blue and the gpt-4 bars are green. This is because aider uses different prompting strategies for GPT-3.5 and 4. GPT-3.5 is only able to reliably edit a file by returning a whole new copy of the file with the edits included. This is the "whole" edit format. GPT-4 is able to use a more efficient "diff" edit format, where it species blocks of code to search and replace. All of this is described and quantified in more detail in the original aider benchmarking writeup: https://aider.chat/docs/benchmarks.html https://aider.chat/docs/benchmarks.html The original article benchmarked both models using both edit formats (and some others). And indeed, gpt-4/whole beats gpt-3.5/whole. But it's very slow and very expensive to ask gpt-4 to return a whole copy of any file that it edits. So it's just much more practical to use the gpt-4/diff, even though it performs a bit worse than gpt-4/whole. Aider will let you do gpt-4/whole if you'd like to spend the time and money: aider --model gpt-4 --edit-format whole Once OpenAI relaxes the rate limits, I will benchmark gpt-4-1106-preview/whole.