3 ms·
Wasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model chang
by andai 3y ago
Wasn't there a study recently that tracked the performance of GPT over time and found significant drop in quality? Did those drops occur at official model changes, or at other times? (i.e. unannounced changes for safety or cost reduction)
I mean, if I was OpenAI, I probably wouldn't make an announcement like "we've just quantized the model and increased our profit margins significantly! The only change on your end will be a slightly dumber model. (Don't worry! Most users won't even notice!)"
- n2d4 3y agoThis one [1]? That tracks two distinct versions (0613 vs. 0314). Also, IMO, the tasks they evaluate aren't useful (I rarely want my LLM to tell me whether 17077 is a prime number), and there's room for cherrypicking/survivorship bias. My guess is that OpenAI did something between 0314 and 0613 that shifted focus away from maths to other subjects. [1] https://arxiv.org/pdf/2307.09009.pdf https://arxiv.org/pdf/2307.09009.pdf
- imdsm 3y agoI haven't tracked the quality but I do track the performance: https://gpt-monitor.adamkdean.co.uk/ https://gpt-monitor.adamkdean.co.uk/ It fluctuates a lot but you can see trends.