4 ms·
It implies our models are wrong. Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound
by kelseyfrog 5y ago
It implies our models are wrong.
Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words.
To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other languages) 181 billion English words - or about 100 times more words than a human will hear by the time they reach adulthood. How is the human brain able to achieve a higher level of success with 1% of the data?
1. https://github.com/openai/gpt-3/blob/master/dataset_statistics/languages_by_word_count.csv https://github.com/openai/gpt-3/blob/master/dataset_statisti...
- nynx 5y agoYeah, this implies backpropagation is deeply suboptimal.
- kelseyfrog 5y agoThat is certainly a possibility. The other (non-mutually exclusive) implications may also be that human language acquisition benefits from being part of a multi-task model. Or that the problem has been overreduced ie: human language acquisition cannot simply be distilled into a words-in->words-out problem and that vision/hearing are actually integral parts of language acquisition that cannot be left out. Or that model arch still has major improvements to be made and attention is not all you need, for example.
- fpgaminer 5y ago> and that vision/hearing are actually integral parts of language acquisition Deaf-blind authors would beg to differ. But yes, a human brain is exposed to lots of other sensory input, and we know from other research that multi-modal models can learn shared representations that benefit from the knowledge of each domain. In Transformer's favor, at least, they are far closer to tabula rasa than the human brain is and likely have to dedicate a lot of their training time to things that are otherwise "baked" into human brains. For example, humans come pre-packaged with V1 and V2 as part of their visual system, but CNNs and ViTs have to learn those filter packages from scratch. I agree with you though. Human brains are able to take single instances of experiences and build a wealth of understanding from them in ways that even modern Transformer architectures are not yet able.
- kristintynski 5y agoIt seems like internal language (thinking in language) is also a way our brains train themselves too? I’ve probably thought 100x more words than I’ve spoken.
- snovv_crash 5y agoThis would map to a sort of semi-supervised approach. For a lot of problems this has shown to drastically reduce the data requirements, but can bump up compute. All those conversations in the shower were actually regularizers!
- Symmetry 5y agoMy understanding is that the binding constraint in training these models is the quantity of computation they consume. While a human makes do with drastically less input data, we also have drastically more computational resources in our heads to work on the problem than Google is using to train its models.
- gwern 5y ago> How is the human brain able to achieve a higher level of success with 1% of the data? The most obvious answer is "the human brain uses a shit-ton more compute", for 18+ years as well. We spend data, which we have in abundance, to save on compute, which we do not. Even at the most generous low-end estimates of the human brain's computing power, we are only barely there; on the high-end estimates that people in love with the ineffable mysteries of the brain love to cite, we are multiple orders of magnitude away from even the biggest supercomputers matching the brain. So no matter which way you slice it, we are extremely compute-poor. Feeding a lot of data through an extremely lightweight optimizer like first-order SGDs is one way to cope with lacking compute: https://www.gwern.net/docs/ai/scaling/2013-bottou.pdf https://www.gwern.net/docs/ai/scaling/2013-bottou.pdf Bottou asks why (even in 2013!) is SGD so hard to dethrone when we can empirically see plenty of optimizers like second-order gradient descent algorithms which can beat SGD quite solidly? His observation is that while they are much better than SGD in terms of iterations or _n_, they lose in compute/wallclock because SGD can just go-brrrr through the data much faster than they can.
- nynx 5y agoYeah, there are ~100B neurons, ~1Q synapses, but how much compute is the brain actually using over time? Some quick googling gives this: - Generation of an action potential seems to use ~2.5×10^−7 J [0] - The brain consumes around 20W during normal activity This seems to imply that there are around 8×10^7, call it 10^8, activations per second [1]. Apparently, the average neuron has 1000 synapses. Let's say each synapse requires 10 mulacc operations per activation. Doing that math gives about 10^12 FLOPs/s [2]. Integrate that over 18 years, and you get roughly 5.7×10^20 FLOPs [3]. PaLM required 2.56×10^24 FLOPs to train [4]. So, we have (way more than) enough compute, we're just not using it efficiently. We're wasting a lot of FLOPs on dense matrix multiplication. There's plenty of wiggle room in these calculations. I checked over the math, but I'd appreciate if someone would let me know if I've missed something. [0]: https://link.springer.com/article/10.1007/s11571-018-9503-3 [1]: https://www.wolframalpha.com/input?i2d=true&i=Divide%5B20+W%2C2.5%E2%80%89%C3%97%E2%80%89Power%5B10%2C%E2%88%927%5D+Joules%5D [2]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C8%5D+Hz+*+1000+*+10+flop [3]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C12%5D+Divide%5BFLOP%2Cs%5D+*+18+years [4]: https://blog.heim.xyz/palm-training-cost/#:~:text=PaLM%20(2022)-,2.5e24,-10x***