3 ms·
The report isn't interested in this question but I am: If we had spent the last decade throwing this amount of resources at some other family of ML techniques,
by abeppu 4y ago
The report isn't interested in this question but I am:
If we had spent the last decade throwing this amount of resources at some other family of ML techniques, would they have shown the same improvement? Or is there some parallel universe where if we had made 500B parameter non-parametric bayesian models or something we would have gotten to a similar place?
- MichaelRazum 4y agoNot an expert but intuitively, let's say in classical stat's. If you add parameters the danger of overfit increases. So it is far from obvious (at least to me) why it works at all and why bigger is obviously better.
- jimmySixDOF 4y agoAnother counterfactual branch could be considered where GPU innovation and market pushes from Gaming and Crypto did not happen to make transformer parallelization such a great value proposition and what alternative paths have not been explored thoroughly as a result.
- riotnrrd 4y agoI'm a deep learning researcher so I can speak to this a little. Until roughly 2010 neural networks were barely used and broadly considered to be less efficient and less accurate than rival techniques (decision trees, support vector machines, etc.). For decades, far more work was put into these other approaches but the amazing thing about neural networks is that, even in a relatively primitive and neglected state, they outperformed traditional techniques. AlexNet is the poster-child of this phenomenon, but look at facial recognition, speech recognition, etc. These are all fields where older approaches are just dead. Nobody is working on SVMs for facial identification or object pose estimation anymore. It's more accurate to say that neural networks succeeded against the research darlings of the time despite being relatively ignored for decades.