4 ms·
That's not true really. With well written prompts GPT4 is better at some things and worse at others than Claude/Llama3. GPT4 only appears to be the best by a
by CuriouslyC 3y ago
That's not true really. With well written prompts GPT4 is better at some things and worse at others than Claude/Llama3. GPT4 only appears to be the best by a wide margin if your benchmark suite is vague, poorly specified prompts and your metric for evaluation is "did it guess what I wanted accurately"
- int_19h 3y agoMy benchmark is giving it novel (i.e. guaranteed to not be in the training set) logical puzzles that require actual reasoning ability, and seeing how it performs. By that benchmark, GPT-4 significantly outperforms both LLaMA 3 and Claude in my personal experience.
- CuriouslyC 3y agoThat's occurring because you're giving it weak prompts, like I said. GPT4 has been trained to do things like chain of thought by default, where as you have to tell Llama/Claude to do some of that stuff. If you update your prompts to suggest reasoning strategies and tell it to perform some chain of thought before hand the difference between models should disappear.
- int_19h 3y agoYou are assuming a great deal of things. No, you can absolutely come up with puzzles where no amount of forced CoT will make the others perform on GPT-4 level. Hell, there are puzzles where you can literally point out where the answer is wrong and ask the model to correct itself, and it will just keep walking in circles making the same mistakes over and over again.