3 ms·
I'm fairly lay but imo GPT-3 demonstrates pretty soundly that huge models are no magic bullet- it's got >2x as many parameters as the human brain has neurons an
by hwillis 6y ago
I'm fairly lay but imo GPT-3 demonstrates pretty soundly that huge models are no magic bullet- it's got >2x as many parameters as the human brain has neurons and it can't do long division. Dogs and other animals get by just fine having less than 1% as many neurons as humans.
Even a billion parameters is a huge model, and a factor of 2.4x increase is not going to make a tremendous difference in your performance. In particular the data-heavy nature of vision stuff means that you'll be bottlenecked by training more than memory, AFAIK (again, lay).
- i-am-curious 6y agoThis is a misleading comparison. You are comparing a massive model with huge models. What you should be comparing are big models vs medium models that a single consumer GPU will fit. And - you don't need to take my word for it, there's tons of papers - the bigger models definitely perform better.
- ipsum2 6y agoModel parameters and biological neurons do not map 1:1.
- benlivengood 6y agoThe human brain has about 100 trillion synapses which is a closer analog to ML model parameters.
- junipertea 6y agoIt can't do long division because it literally can't read the number inputs, due to the way text is encoded (BPE). That it manages to learn any sort of arithmetic despite that is pretty impressive to me.