3 ms·
Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgB
by huijzer 3y ago
Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy.
[1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA
- py4 3y agoThis. We have not exhausted all the techniques at our disposal yet. We do need to look for a new architecture though, but these are orthogonal
- scriptsmith 3y agoSeems like he actually disagrees here: If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer, and we are very confident we are going to get a better result. https://youtu.be/zjkBMFhNj_g?t=1543 https://youtu.be/zjkBMFhNj_g?t=1543 (23:43)
- huijzer 3y agoAnd then at 35 minutes he spends a few minutes talking about ideas for algorithmic improvements.