4 ms·

What will the benefit be of more expensive models? More facts, because it's consumed more information? More ability to, say, adjust writing style? Or is this al

What will the benefit be of more expensive models? More facts, because it's consumed more information? More ability to, say, adjust writing style? Or is this all necessary just to filter out the garbage recycled AI content it's now consuming?

jiggawatts 2y ago
Right around the time GPT-4 was first announced, OpenAI published a paper that basically said that training can "just keep going" with no obvious end in sight. Recently, Meta tried to train a model 75x as long as is naively optimal, and it just kept getting better. Better in this case means some combination of "less errors for the same size" and/or "bigger and smarter". Fundamentally, they're still the same thing, just more and better. Unfortunately, the scaling is (roughly) logarithmic. So for every 10x increase in scale you get a +1 better model. Scaling up 1,000x gets you just a +3 improvement, and so on.
- iamleppert 2y ago
  And what, exactly is the ROI on "better"? Who cares if the model is better, is it 100B better? Who are going to buy these services, what consumer will pay for it?
  - jiggawatts 2y ago
    Meta has something like half a million modern GPUs that they’ve purchased outright. They can afford to keep training models “forever”. This is useful to eke out every last drop of quality per gigabyte of model file size. It also keeps the models up to date with current events. Obviously this scaling becomes too inefficient at infinite scale not just because of training costs that’ll never be recouped but also increasing inference cost with larger models. Some fundamentally new architectures will need to be developed to take much better advantage of increased computer power. I suspect the major players are investing in hardware now in the hope that some revolutionary new algorithm is invented soon and they’ll be ready for it. It’s… a bit of a gamble!
lm28469 2y ago
> What will the benefit be of more expensive models? Bleed investors dry before the next fad pops up
- blitzar 2y ago
  > What will the benefit be of more expensive models? A G650 to fly to your 85m yacht in the med doesnt come cheap.

jiggawatts 2y ago

Right around the time GPT-4 was first announced, OpenAI published a paper that basically said that training can "just keep going" with no obvious end in sight. Recently, Meta tried to train a model 75x as long as is naively optimal, and it just kept getting better. Better in this case means some combination of "less errors for the same size" and/or "bigger and smarter". Fundamentally, they're still the same thing, just more and better. Unfortunately, the scaling is (roughly) logarithmic. So for every 10x increase in scale you get a +1 better model. Scaling up 1,000x gets you just a +3 improvement, and so on.

iamleppert 2y ago
And what, exactly is the ROI on "better"? Who cares if the model is better, is it 100B better? Who are going to buy these services, what consumer will pay for it?
- jiggawatts 2y ago
  Meta has something like half a million modern GPUs that they’ve purchased outright. They can afford to keep training models “forever”. This is useful to eke out every last drop of quality per gigabyte of model file size. It also keeps the models up to date with current events. Obviously this scaling becomes too inefficient at infinite scale not just because of training costs that’ll never be recouped but also increasing inference cost with larger models. Some fundamentally new architectures will need to be developed to take much better advantage of increased computer power. I suspect the major players are investing in hardware now in the hope that some revolutionary new algorithm is invented soon and they’ll be ready for it. It’s… a bit of a gamble!

lm28469 2y ago

> What will the benefit be of more expensive models? Bleed investors dry before the next fad pops up

blitzar 2y ago
> What will the benefit be of more expensive models? A G650 to fly to your 85m yacht in the med doesnt come cheap.