4 ms·

What isn't really discussed here (and I wish was!) is the corollary to the price scaling of successive generations: the outcome isn't proportionate to the incre

by lsy 3y ago

What isn't really discussed here (and I wish was!) is the corollary to the price scaling of successive generations: the outcome isn't proportionate to the increased cost. GPT-4 cost thirty times as much as GPT-3 to train, but it isn't thirty times better. It's not even twice as good! I suspect the majority of people using an OpenAI product are still using GPT-3.5 in the free tier and are mostly satisfied with it, because the basic proposition — plausibly fluent English text — is still there, and GPT-4's improvements aren't needed for most of the use cases an LLM satisfies. Following this out, you have the prospect of this company burning multiple trillion dollars to create a foundation model that is slower than GPT-4 and possibly only 5-10% better. The only way this makes any sense financially is if you think that extra 5-10% takes us to some kind of AGI that solves all of the reasons GPT-4 hasn't put everyone out of a job. I suspect OpenAI doesn't think this, but maybe "training another model" is easier to pitch than whatever the process would be to find product-market fit for their current model.

atleastoptimal 3y ago
> GPT-4 cost thirty times as much as GPT-3 to train, but it isn't thirty times better. It's not even twice as good! Any small percentage gain in general intelligence is well worth it given the huge implications. If you believe AGI is possible, and it certainly makes sense to believe that given the performance improvements in foundation models over the past 5 years, then even an inordinately large sum justifies getting there. But more practically. Let's say GPT 4 is not good enough to completely replace any human. Let's say GPT-5's improvements allow it to replace the bottom 10% of white collar jobs. That alone, discounting the productivity improvements for people who use it but aren't replaced by it, would offset the even enormous upfront costs.
- dns_snek 3y ago
  You're just using wishful assumptions and circular reasoning to promote a silly investment ("Let's assume our investment pays off, that alone makes it worthwhile to invest in."). You could just as easily spend trillions and end up with less than 20% improvement over GPT-4. Future gains are not guaranteed.
  - atleastoptimal 3y ago
    > Future gains are not guaranteed. That's why OpenAI and others have put a lot of research into predictive scaling laws, different architectures, etc. Their confidence is bolstered by an understanding of the limits of what scaling alone can do, but it's clear there are avenues that haven't been explored yet, given that every largest chat model is the best chat model by a predictable margin.
- tivert 3y ago
  > Let's say GPT 4 is not good enough to completely replace any human. Let's say GPT-5's improvements allow it to replace the bottom 10% of white collar jobs. That alone, discounting the productivity improvements for people who use it but aren't replaced by it, would offset the even enormous upfront costs. You're also forgetting the positive climate impact of eliminating those people's CO2 production. /s

by lsy 3y ago

atleastoptimal 3y ago

> GPT-4 cost thirty times as much as GPT-3 to train, but it isn't thirty times better. It's not even twice as good! Any small percentage gain in general intelligence is well worth it given the huge implications. If you believe AGI is possible, and it certainly makes sense to believe that given the performance improvements in foundation models over the past 5 years, then even an inordinately large sum justifies getting there. But more practically. Let's say GPT 4 is not good enough to completely replace any human. Let's say GPT-5's improvements allow it to replace the bottom 10% of white collar jobs. That alone, discounting the productivity improvements for people who use it but aren't replaced by it, would offset the even enormous upfront costs.

dns_snek 3y ago
You're just using wishful assumptions and circular reasoning to promote a silly investment ("Let's assume our investment pays off, that alone makes it worthwhile to invest in."). You could just as easily spend trillions and end up with less than 20% improvement over GPT-4. Future gains are not guaranteed.
- atleastoptimal 3y ago
  > Future gains are not guaranteed. That's why OpenAI and others have put a lot of research into predictive scaling laws, different architectures, etc. Their confidence is bolstered by an understanding of the limits of what scaling alone can do, but it's clear there are avenues that haven't been explored yet, given that every largest chat model is the best chat model by a predictable margin.
tivert 3y ago
> Let's say GPT 4 is not good enough to completely replace any human. Let's say GPT-5's improvements allow it to replace the bottom 10% of white collar jobs. That alone, discounting the productivity improvements for people who use it but aren't replaced by it, would offset the even enormous upfront costs. You're also forgetting the positive climate impact of eliminating those people's CO2 production. /s