4 ms·
> I'd say with GPT4 they probably reached the limit of how big the dataset can be I’m curious about this too; not just on the dataset size, but also the model
by brendamn 3y ago
> I'd say with GPT4 they probably reached the limit of how big the dataset can be
I’m curious about this too; not just on the dataset size, but also the model size. My hunch is that the rapid improvements of the underlying model by making it bigger/giving it more data will slow, and there’ll be more focus on shrinking the models/other optimisations.
- int_19h 3y agoI don't think we're anywhere close to the limit of sheer hardware scalability on this. Returns are diminishing, but if GPT-4 (with its 8+ k context window) is any indication, even those diminishing returns are still very worthwhile. If anything, I wonder if the actual limit that'll be hit first will be the global manufacturing capacity for relevant hardware. Check out the stock price of NVDA since last October.