3 ms·
I'm not convinced we're going to get much further than this, at least without a see change in how this stuff is done. We saw a pretty big amount of diminishing
by __loam 3y ago
I'm not convinced we're going to get much further than this, at least without a see change in how this stuff is done. We saw a pretty big amount of diminishing returns with GPT-4, which cost 2 orders of magnitude more money to train than 3.5. Is the performance more than 2 orders of magnitude better than 3.5?
How much compute is too much to be spending on this bullshit. 30% of humanity's total computing resources? 70%?
GPT was a big innovation in parallelizing the training process. I think the optimism that we're on the beginning of a sigmoid curve here ought to be questioned.
- danenania 3y agoThe performance doesn't have to scale with the training cost. It's ok if we have diminishing returns. The only question is if we can keep getting improvements. If we could get a 10x improvement over GPT-4 (while maintaining safety/alignment), the potential benefits to productivity across the entire world economy are so great that it justifies nearly unlimited investment.
- __loam 3y agoThis is delusional.
- deleted 3y ago[deleted]