3 ms·
In my intuition it makes sense that there is going to be some significant friction in LLM development going forward. We're talking about models that will cost u
by captainbland 2y ago
In my intuition it makes sense that there is going to be some significant friction in LLM development going forward. We're talking about models that will cost upwards of $1bn to train. Save for a technological breakthrough, GPT-6/7 will probably have to wait for hardware to catch up.
- rrrrrrrrrrrryan 2y agoI think the main bottleneck right now is training data - they've basically exhausted all public sources of data, so they have to either pay humans to generate new data from scratch or pay for the reasoning models to generate (less useful) synthetic training data. The next bottleneck is hardware, and the least important bottleneck is money.
- energy123 2y agoIt's more that you have to allocate the compute the right way. Noam Brown's analogy is, you could train a massive one shot foundation model to predict the next best Go move, but that would be stupid. Better to use some test time search. You get better results for less money. Same is happening in LLMs.