3 ms·
> But at what point do we run out of training data? Multi-modal LLMs trained on video / YouTube won't run out of data any time soon. And LLMs can always train
by pton_xd 2y ago
> But at what point do we run out of training data?
Multi-modal LLMs trained on video / YouTube won't run out of data any time soon. And LLMs can always train on an endless supply of synthetic data like code.
> And at what point does spending $x million more on training the base models no longer provide that much more value?
My understanding is we're still waiting to see where on the S-curve LLMs are in regards to scaling. So far it seems like scaling hasn't hit any diminishing returns.
I haven't read many sources talking about the practicality of that though. Even if LLMs continue scaling performance as we throw more compute + data at them, at some point it may just become uneconomical to continue. Capital expenditures also have their own utility curve. How far away are we from that point?
- jacobr1 2y ago> Capital expenditures also have their own utility curve This a function of the use-cases. For simple use cases like document summarization, or speech-to-text, we might already be there. There are already applications where using something like gpt-3-turbo or a local-llm vs gpt-4 make more economic sense. The marginal improvement in reasoning or extra knowledge don't provide benefits relative to the inference costs. And you can see this playing out with Mistral, Anthropic and Google also providing models at different cost/complexity tradeoffs. But .... if we get _new_ use cases, emergent use cases, from bigger models, there will be continued value in chasing that frontier. Reasoning is still poor in many areas. The ability to learn outside of application specific memory patterns that store state is largely unexplored. We are still far from AGI and there is a ton of value to be minded between current systems and a superhuman capabilities