4 ms·

GPT-4 is 1.8T parameters. Until a model with at least 2-4x the number of params with no discernable improvement is released can we say that improvement has stal

GPT-4 is 1.8T parameters. Until a model with at least 2-4x the number of params with no discernable improvement is released can we say that improvement has stalled. The biggest I imagine are Opus and Gemini Ultra with around 2T, far less that the geometric increase to demonstrate new emergent capabilities. Either way evidence has not been put forward yet to disprove scaling laws so until then, the apparent stalling is simply that a next-gen OOM model has yet to be trained and deployed.

gnfedhjmm2 2y ago
The whole “parameters” thing is just a marketing term though, if they wanted to say they use a babillion parameters how would that change anything? The term is subjective and open to interpretation.
- atleastoptimal 2y ago
  Parameters is the literal size of the model. How is it a marketing term? Is saying that a computer has 16GB of ram just a marketing term?
  - hurrdurr57 2y ago
    Well, the statement that GPT-4 is 1.8T parameters is a little misleading since it's really a 8 x 220B MoE (according to the rumors at least). Also the size of the model itself isn't the only factor that determines performance, LLama 3 70B outperforms LLama 2 70B even though they have the same size.
dhjhydefh 2y ago
It's well past time to speak of emergent capabilities. That idea was debunked by this paper. It was given a Best Paper award and was one of the few oral presentations. It's time to move on. https://openreview.net/forum?id=ITw9edRDlD https://openreview.net/forum?id=ITw9edRDlD
- atleastoptimal 2y ago
  One paper attempting one hypothesis against one model doesn't completely debunk emergent capabilities. If you've seen one of the thousands of novel capabilities of GPT-4 level models you'd realize this.
- pizza 2y ago
  Eh, I don't think that's their position: > nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities; rather, our message is that some previously claimed emergent abilities appear to be mirages induced by researcher analyses imo their paper's position is - downstream task metric jumps in accuracy don't imply sudden changes in pretrain loss - you can cherry pick metrics to show the sudden appearance of emergent abilities - model scale is not enough to explain emergent abilities because you can scale up one model family and not get them Their follow-up paper [0] has an appendix that touches upon how: - it's not impossible for continuous metrics to undergo sudden increases of performance - therefore not all sudden increases of performance are a consequence of using discontinuous metrics - therefore emergent abilities could still actually exist [0] Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? https://arxiv.org/abs/2406.04391 https://arxiv.org/abs/2406.04391

gnfedhjmm2 2y ago

The whole “parameters” thing is just a marketing term though, if they wanted to say they use a babillion parameters how would that change anything? The term is subjective and open to interpretation.

atleastoptimal 2y ago
Parameters is the literal size of the model. How is it a marketing term? Is saying that a computer has 16GB of ram just a marketing term?
- hurrdurr57 2y ago
  Well, the statement that GPT-4 is 1.8T parameters is a little misleading since it's really a 8 x 220B MoE (according to the rumors at least). Also the size of the model itself isn't the only factor that determines performance, LLama 3 70B outperforms LLama 2 70B even though they have the same size.

dhjhydefh 2y ago

It's well past time to speak of emergent capabilities. That idea was debunked by this paper. It was given a Best Paper award and was one of the few oral presentations. It's time to move on. https://openreview.net/forum?id=ITw9edRDlD https://openreview.net/forum?id=ITw9edRDlD

atleastoptimal 2y ago
One paper attempting one hypothesis against one model doesn't completely debunk emergent capabilities. If you've seen one of the thousands of novel capabilities of GPT-4 level models you'd realize this.
pizza 2y ago
Eh, I don't think that's their position: > nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities; rather, our message is that some previously claimed emergent abilities appear to be mirages induced by researcher analyses imo their paper's position is - downstream task metric jumps in accuracy don't imply sudden changes in pretrain loss - you can cherry pick metrics to show the sudden appearance of emergent abilities - model scale is not enough to explain emergent abilities because you can scale up one model family and not get them Their follow-up paper [0] has an appendix that touches upon how: - it's not impossible for continuous metrics to undergo sudden increases of performance - therefore not all sudden increases of performance are a consequence of using discontinuous metrics - therefore emergent abilities could still actually exist [0] Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? https://arxiv.org/abs/2406.04391 https://arxiv.org/abs/2406.04391