4 ms·
This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the lite
by cs702 6y ago
This looks like a big deal to me:
1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain [a], but consider this: Models with trillions of weights are suddenly looking... achievable.
2. The model achieves competitive results on many NLP tasks and benchmarks WITHOUT FINETUNING. Let me repeat that: there is no finetuning. There is only unsupervised (i.e., autoregressive) pretraining. For each downstream NLP task or benchmark, the pretrained model is given text instructions, and possibly sample text with questions and answers. The NLP tasks on which the model was tested include translation, question-answering, cloze tasks, unscrambling words, using novel words in sentences, and performing 3-digit arithmetic.
3. The model is tested only in a ZERO-SHOT or FEW-SHOT setting. In other words, for each NLP task, the pretrained model is given text instructions with zero examples, or text instructions with a small number of examples (typically 10 to 100). As with human beings, GPT-3-173B doesn't need lots of examples to perform competitively in novel NLP tasks.
4. The results reported by this paper on all NLP tasks and benchmarks should be seen as a BASELINE. These results likely could be meaningfully improved with conventional finetuning.
5. The model’s text generation FOOLS HUMAN BEINGS, without having to cherry-pick examples.
--
[a] https://www.google.com/search?q=number+of+synapses+in+human+brain https://www.google.com/search?q=number+of+synapses+in+human+...
- taneq 6y ago> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.
- safog 6y agoI remember reading about fruit fly brains and how we're at a point where we can computationally simulate them now, but I'm not sure where that went. Anyone know?
- ricardorivaldo 6y agoTake a look at http://fruitflybrain.org/ http://fruitflybrain.org/
- canjobear 6y agoIt's not clear whether a parameter in a neural network maps cleanly onto a synapse in a biological brain.
- gambler 6y agoI think it's becoming pretty clear that they don't. First, scientists uncovered many additional ways neurons interact with one another[1]. Second, it seems that individual neurons do way more computing than in the simplistic ANN models [2]. [1]: https://en.wikipedia.org/wiki/Ephaptic_coupling https://en.wikipedia.org/wiki/Ephaptic_coupling [2]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5740076/l https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5740076/l
- ibeckermayer 6y agoIt’s very clear that they don’t.
- skdotdan 6y agoAFAIK they used half-precision (Float16)
- cs702 6y agoThanks. I should have written "if using Float32," which is what I meant -- instead of "with Float32," which in hindsight reads a bit ambiguous. But regardless of which floating-point representation is used, the number of weights is still in the hundreds of billions... which is insane.
- alphagrep12345 6y agoWhat do you mean by text instructions? If I want to translate a sentence, would I just feed in model - translate "Hello world"?
- Traktor 6y agoSee page 7 of the paper. You give the model instruction such as "Translate from X to Y" then you pass examples (if you go for few shot) followed by the sentence you want to translate.
- gambler 6y agoI'll wait for a working interactive model before blindly believing these statements. GPT-2 was hyped through the roof, but when inspected with a bit of criticality it demonstrated glitches that told us more about how it actually works than "good" examples: https://medium.com/@VictorBanev/interrogating-full-gpt-2-10ae1a9179f6 https://medium.com/@VictorBanev/interrogating-full-gpt-2-10a... ML models should be pushed to their limit, because that's where you gather most useful information about what they actually do. Their results need to be critically examined with both exploratory and hypothesis-driven testing. And yet this is never done in initial papers and rarely done afterwards. What was the last AI paper you've read that that said "and here is a list of things out model failed at"?
- gwern 6y agoThat's a very sloppy post. He does a single example, not even running locally or changing sampling parameters, and then concludes that GPT-2 is doing nothing but pattern-matching? A lot of people underestimate NNs because the sampling from them (top-k! how much dumber and cruder can you get? nucleus works better, but is still obviously suboptimal) destroys a lot of dark knowledge. I noticed this with Gary Marcus's claims about GPT-2 too: he would try once, without changing any sampling settings, and conclude that it wasn't doing anything, but if you tried, you would get different results. I'm not the only one to notice that: https://www.quantamagazine.org/common-sense-comes-to-computers-20200430/ https://www.quantamagazine.org/common-sense-comes-to-compute... Such tests can prove the presence of knowledge, but not the absence... And of course, GPT-3 does extensive arithmetic tricks: https://arxiv.org/pdf/2005.14165.pdf#page=22 https://arxiv.org/pdf/2005.14165.pdf#page=22
- gambler 6y agoThe article I linked to makes claims only about the model it tests and since it actually links to an online implementation, anyone can try to reproduce the results and see for themselves. This is more than I can say about most chatter about ML. >Such tests can prove the presence of knowledge, but not the absence... This sounds like a setup for non-falsifiable beliefs.