5 ms·
The biggest takeaway from the article for me was GPT 4 has 100 trillion parameters, 500x more then GPT 3. that type of exponential scaling is going to hit an up
by thefreeman 4y ago
The biggest takeaway from the article for me was GPT 4 has 100 trillion parameters, 500x more then GPT 3. that type of exponential scaling is going to hit an upper bound really quickly. So when you say it “moves further with each update” you should realize that the improvements are coming from throwing massively more scale at the problem and not some underlying improvement in the technique, and also balance the amount of improvement with the scale itself.
obviously gpt-4 is nowhere near 500x better than gpt-3. let’s say it’s 20% better (very generous imo). can they realistically 500x the model again? and if so, is that going to be worth an additional 4% gain to the original model quality? numbers are completely made up and math is probably wrong but i think i’m hopefully making my point, that diminishing returns will quickly become a blocker with this type of scaling.
- carbocation 4y agoI would hesitate to assume that is true. The 100 trillion param number has not been confirmed by anyone who could know if it’s true.
- cinntaile 4y agoWe shouldn't forget that at the same time effort is being spent to achieve the same results with a lot fewer parameters.
- famouswaffles 4y agoGpt-4 doesn't have 100 trillion parameters. It's not much larger than 3. It just has a lot more data according to Altman.
- qualudeheart 4y agoAltman himself has said it doesn’t have 100T parameters. It has more data in line with the Chinchilla laws.
- blueorange8 4y agoI don't believe they spent 7 months just "making gpt-4 safer" - I think they spent a very long time doing human reinforcement learning to make it better and now hope to speed that process up the next time using gpt-4 itself as the reinforcement.
- r0uv3n 4y agoAFAIK the paper mentioned that RLHF mostly decreased its capabilities. It seems more likely to me that just longer normal training was one of the main reason for the increased capabilities.