3 ms·
> GPT-4 or another huge language model like it will be released, and we will see diminishing returns continue to plateau. It was my understanding that GPT-3 sh
by oezi 5y ago
> GPT-4 or another huge language model like it will be released, and we will see diminishing returns continue to plateau.
It was my understanding that GPT-3 showed pretty well that the returns are not dimishing but rather surprisingly still scaling along with data rather nicely.
- KhoomeiK 5y agoFigure 3.8 in the GPT-3 paper (SuperGLUE Performance) pretty clearly shows a plateau in performance, with the increase in performance from 13B to 175B parameters (13.5x params, +6 score) not being significantly different from the increase between 6.7B to 13B (1.9x params, +4 score).