2 ms·
A lot of people are saying that GPT-3 is a huge step up from GPT-2. This is confusing to me looking at the results in the paper as someone not too familiar with
by highfrequency 6y ago
A lot of people are saying that GPT-3 is a huge step up from GPT-2. This is confusing to me looking at the results in the paper as someone not too familiar with GPT-2.
The tasks where GPT-3 scores much higher than GPT-2 are the ones most removed from broad language comprehension / general intelligence: arithmetic and unscrambling tasks (as in: which character does not belong in the string "app$le"). On the other tasks it does 5-15% better. This seems like a modest improvement for a model that is thirteen times larger than its predecessor.
I don't mean to minimize the accomplishment--training models at such a massive scale is difficult. But genuinely wondering if I am missing something here--are people's reactions more from a qualitative judgement about the paragraphs produced by GPT-3 vs. GPT-2? If so, a non-cherry-picked side-to-side comparison would be great to see. If this is a big leap, why doesn't it show up the quantitative results?
- dj_mc_merlin 6y agoThe important thing is that there is qualitative improvement only from quantitive increase. Other software doesn't function like this, switching to a better CPU won't enable new features in Excel. This is not a given, and the fact that there is some proof that it could happen is significant, although it's too early and unsure to really tell. We are nowhere near the border of the amount of resources we can put into a bigger model, what if it just keeps getting better?