3 ms·
When were the GPT-4 benchmarks calculated, on original release or more recently? (curious per the debate about alleged gpt-4 nerfing)
by ineedasername 3y ago
When were the GPT-4 benchmarks calculated, on original release or more recently? (curious per the debate about alleged gpt-4 nerfing)
- lappa 3y agoThey're based on the original technical report. "Refuel" has run a different set of benchmarks on GPT-3.5 and GPT-4 and found a decline in quality. https://www.refuel.ai/blog-posts/gpt-3-5-turbo-model-comparison https://www.refuel.ai/blog-posts/gpt-3-5-turbo-model-compari...
- ShamelessC 3y agoPlenty of the complaints/accusations predate the release of the 0613 set of models. To be clear, I have trouble with the theory as I have not yet seen evidence of "nerfing". What you provided is actually the _only_ evidence I've seen that suggests degradation - but in this case OpenAI is being completely transparent about it and allows you to switch to the 0314 model if you would like to. Every complaint I have seen has been highly anecdotal, lacking any rigor, and I bet are explained by prolonged usage resulting in noticing more errors. Also probably a bit of "the magic is gone now" psychological effect (like how a "cutting edge" video game such as Half-Life 2 feels a bit lackluster these days).