4 ms·

And the industry is seeing strong downward pressure on model size. I don’t see this as good evidence that model parameter increases have reached a limit. I se

And the industry is seeing strong downward pressure on model size. I don’t see this as good evidence that model parameter increases have reached a limit. I see it as evidence that compute costs are very high.

jakderrida 2y ago
Exactly what I was thinking. He also says >Paradoxically, smaller models require more training to reach the same level of performance. It's not a paradox at all. If less training to reach the same level of performance was true, that would be a paradox whereby they'd be trained for under a nanosecond to achieve optimal performance/size payoff.
__loam 2y ago
GPT-4 offered a marginal improvement over the capability of GPT-3. Depending on how you measure it, it would be hard to call it an exponential improvement. Let's say 30% just to put a number on things. A significant improvement. GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost? That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.
- zwaps 2y ago
  the improvement of gpt4 over even 3.5 is significant and for practical applications these improvements exceed what is indicated by benchmarks There are many things which earlier models maybe somewhat did, but only large models do reliably to the point of being usable for more than tech demos
  - Pat_Murph 2y ago
    Can you give examples of practical applications that have been significantly improved?
- ithkuil 2y ago
  The disconnect between benchmarks and practical use is interesting. I guess it's hard to build accurate benchmarks. I find gpt-4 and gpt-3 a world apart. Perhaps it's just that it crossed a tipping point that made it useful for _me_ and the next 10x investment in training wouldn't achieve the same effect for me (but perhaps for somebody else?)

jakderrida 2y ago

Exactly what I was thinking. He also says >Paradoxically, smaller models require more training to reach the same level of performance. It's not a paradox at all. If less training to reach the same level of performance was true, that would be a paradox whereby they'd be trained for under a nanosecond to achieve optimal performance/size payoff.

__loam 2y ago

GPT-4 offered a marginal improvement over the capability of GPT-3. Depending on how you measure it, it would be hard to call it an exponential improvement. Let's say 30% just to put a number on things. A significant improvement. GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost? That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.

zwaps 2y ago
the improvement of gpt4 over even 3.5 is significant and for practical applications these improvements exceed what is indicated by benchmarks There are many things which earlier models maybe somewhat did, but only large models do reliably to the point of being usable for more than tech demos
- Pat_Murph 2y ago
  Can you give examples of practical applications that have been significantly improved?
ithkuil 2y ago
The disconnect between benchmarks and practical use is interesting. I guess it's hard to build accurate benchmarks. I find gpt-4 and gpt-3 a world apart. Perhaps it's just that it crossed a tipping point that made it useful for _me_ and the next 10x investment in training wouldn't achieve the same effect for me (but perhaps for somebody else?)