5 ms·

This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively large

by cs702 8y ago

This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performance of a Transformer in the tested tasks improves log-linearly with the number of parameters, suggesting that even the largest model tested, with 1.5B parameters, still underfits 40GB of text. This is compelling evidence that we do NOT need new architectures, NOR new kinds of training objectives, NOR new theories, for better language modeling! We can get better language modeling simply by increasing model capacity (i.e., by adding more parameters to existing models), which becomes easier and simpler to do as hardware continues to improve over time. Great work. PS. In case it's not clear: I'm not saying we should suddenly stop searching for new, better ideas and architectures. That would be silly. Please don't attack a straw-man :-) [a] https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762

modeless 8y ago
Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Humans don't need to read through 40 GB of text multiple times to learn to write.
- fartcannon 8y ago
  I guess the data requirements act as a stand-in for our genetic evolution... Our brain models are good at learning the things we learn. Our computer models aren't yet. /tautology
- cs702 8y ago
  > Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Yes, agreed. Nothing I said above contradicts that! :-) > Humans don't need to read through 40 GB of text multiple times to learn to write. Yes, that's true... but to keep the comparison fair, note that we do need many years of schooling to learn to read, say, at a high-school or college level. And before learning to read, we first must learn to speak, which surely helps. And we also get to inhabit bodies that see, smell, touch, and interact with the physical objects that we read and speak about during our formative years, which also helps. The more one thinks about it, 40GB of data is actually a tiny figure in comparison to the amount of training data that flows continuously to our brain from all senses. I think I read once that our brains process on the order of 10 to 100 GB of training data per second.
  - Cybiote 8y ago
    Conscious processes are estimated to work on the order of 10^2 bits. Vision, at the retina, is estimated at 10^7 bits/sec. It drops another order of magnitude by V1. Also note that long as they're not isolated, a deaf and blind person has no trouble getting to full human reasoning ability despite being vastly more impoverished in available data compared to the average person. A human will also be learning vision, hearing, walking, physics, causal reasoning, and much more. This comparison just isn't well grounded. Task specific is how much training does a young brain require to learn to produce language? If the brain comes with innate advantages then rather than be resort to inefficiency and excusing our models, we should try to see if they can be bettered.
    - EForEndeavour 8y ago
      I'm not well versed at all in signal theory, so I'm genuinely curious how these bitrate estimates are made, and would love to see the source of these specific numbers. How do you estimate the effective (digital) bitrate of an inherently analogue system?
  - pmontra 8y ago
    After 40 GB of text the model doesn't know anything about how the world works, and it shows many times in the examples. Nobody would do some of those mistakes, not even young kids. Other mistakes are more subtle but still show a total lack of understanding. Then yes, it's enough to write text that nobody really cares about and that could cover a lot of what we read.
    - riku_iki 8y ago
      It's because nobody dumps 40GB on kid. Kids go through long process of feedback-corrections. I imagine if there would be some crowd funded project to provide feedback about mistakes to this model, it would learn and produce better results fast.
- sanxiyn 8y ago
  40 GB is surprisingly close. I estimate I already read at least 4 GB of text so far. That's just 10 times more samples. I probably write better than GPT-2, but certainly not faster.
  - defen 8y ago
    4GB of English? 500 words per single-spaced page, 5 letters on average per English word, so 2500 bytes/page. 4 gigs then would be 1,600,000 pages. That's 219 single-spaced pages per day every single day for 20 years straight. High, but I guess not outside the realm of possibility depending on the complexity of the text.
    - naniwaduni 8y ago
      For perspective, 219 pages amounts to about a railway novel in a day, or about two ~800-page fantasy doorstoppers in a week. At a rule of thumb of about a page per minute for light reading, it's just under four hours: a hefty investment, but well within the realm of possibility. High compared to the general population, but table stakes for a book club or otherwise avid reader. I'd've easily doubled or tripled that over the summer as a teenager.
fartcannon 8y ago
I feel like my human brain underfits a lot of data. My human brain definite zero-shot transfer learns almost everything.
rosstaylor90 8y ago
I think your conclusion is too strong. Yes, we know bigger models and data generally lead to better performance (e.g. BigGAN results last year) but progress in architecture can still speed up progress in ML tasks. If we were still stuck using RNNs and LSTMs for language modeling we wouldn't be talking about this news today.
deleted 8y ago
[deleted]
ninjin 8y ago
> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible. [1]: http://www.aclweb.org/anthology/D/D16/D16-1244.pdf http://www.aclweb.org/anthology/D/D16/D16-1244.pdf [2]: http://papers.nips.cc/paper/7181-attention-is-all-you-need http://papers.nips.cc/paper/7181-attention-is-all-you-need [3]: https://arxiv.org/abs/1301.3781 https://arxiv.org/abs/1301.3781 I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.
- cs702 8y ago
  The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)
  - ninjin 8y ago
    I do not get it. If what you are trying to say is: “more data/computation helps”, then why even bother throwing in models, training objectives, and theory into your statement? We have known that more data helps for at least two decades, so saying this is stating the obvious. Are you then trying to say: “Right at this very moment we can push a bit further with more data/computation given what we have for this specific task”, well, then again this is nothing particularly insightful to bring to the table as surely this occurs at least once a month for a given task/model category. This is not a matter of clarity, it is a matter of form – drop the hyperbole and speak specifics rather than trying to appease both sides of a fictitious divide.
- Cybiote 8y ago

by cs702 8y ago

modeless 8y ago

Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Humans don't need to read through 40 GB of text multiple times to learn to write.

fartcannon 8y ago
I guess the data requirements act as a stand-in for our genetic evolution... Our brain models are good at learning the things we learn. Our computer models aren't yet. /tautology
cs702 8y ago
> Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Yes, agreed. Nothing I said above contradicts that! :-) > Humans don't need to read through 40 GB of text multiple times to learn to write. Yes, that's true... but to keep the comparison fair, note that we do need many years of schooling to learn to read, say, at a high-school or college level. And before learning to read, we first must learn to speak, which surely helps. And we also get to inhabit bodies that see, smell, touch, and interact with the physical objects that we read and speak about during our formative years, which also helps. The more one thinks about it, 40GB of data is actually a tiny figure in comparison to the amount of training data that flows continuously to our brain from all senses. I think I read once that our brains process on the order of 10 to 100 GB of training data per second.
- Cybiote 8y ago
  Conscious processes are estimated to work on the order of 10^2 bits. Vision, at the retina, is estimated at 10^7 bits/sec. It drops another order of magnitude by V1. Also note that long as they're not isolated, a deaf and blind person has no trouble getting to full human reasoning ability despite being vastly more impoverished in available data compared to the average person. A human will also be learning vision, hearing, walking, physics, causal reasoning, and much more. This comparison just isn't well grounded. Task specific is how much training does a young brain require to learn to produce language? If the brain comes with innate advantages then rather than be resort to inefficiency and excusing our models, we should try to see if they can be bettered.
  - EForEndeavour 8y ago
    I'm not well versed at all in signal theory, so I'm genuinely curious how these bitrate estimates are made, and would love to see the source of these specific numbers. How do you estimate the effective (digital) bitrate of an inherently analogue system?
- pmontra 8y ago
  After 40 GB of text the model doesn't know anything about how the world works, and it shows many times in the examples. Nobody would do some of those mistakes, not even young kids. Other mistakes are more subtle but still show a total lack of understanding. Then yes, it's enough to write text that nobody really cares about and that could cover a lot of what we read.
  - riku_iki 8y ago
    It's because nobody dumps 40GB on kid. Kids go through long process of feedback-corrections. I imagine if there would be some crowd funded project to provide feedback about mistakes to this model, it would learn and produce better results fast.
sanxiyn 8y ago
40 GB is surprisingly close. I estimate I already read at least 4 GB of text so far. That's just 10 times more samples. I probably write better than GPT-2, but certainly not faster.
- defen 8y ago
  4GB of English? 500 words per single-spaced page, 5 letters on average per English word, so 2500 bytes/page. 4 gigs then would be 1,600,000 pages. That's 219 single-spaced pages per day every single day for 20 years straight. High, but I guess not outside the realm of possibility depending on the complexity of the text.
  - naniwaduni 8y ago
    For perspective, 219 pages amounts to about a railway novel in a day, or about two ~800-page fantasy doorstoppers in a week. At a rule of thumb of about a page per minute for light reading, it's just under four hours: a hefty investment, but well within the realm of possibility. High compared to the general population, but table stakes for a book club or otherwise avid reader. I'd've easily doubled or tripled that over the summer as a teenager.

fartcannon 8y ago

I feel like my human brain underfits a lot of data. My human brain definite zero-shot transfer learns almost everything.

rosstaylor90 8y ago

I think your conclusion is too strong. Yes, we know bigger models and data generally lead to better performance (e.g. BigGAN results last year) but progress in architecture can still speed up progress in ML tasks. If we were still stuck using RNNs and LSTMs for language modeling we wouldn't be talking about this news today.

deleted 8y ago

[deleted]

ninjin 8y ago

> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible. [1]: http://www.aclweb.org/anthology/D/D16/D16-1244.pdf http://www.aclweb.org/anthology/D/D16/D16-1244.pdf [2]: http://papers.nips.cc/paper/7181-attention-is-all-you-need http://papers.nips.cc/paper/7181-attention-is-all-you-need [3]: https://arxiv.org/abs/1301.3781 https://arxiv.org/abs/1301.3781 I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.

cs702 8y ago
The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)
- ninjin 8y ago
  I do not get it. If what you are trying to say is: “more data/computation helps”, then why even bother throwing in models, training objectives, and theory into your statement? We have known that more data helps for at least two decades, so saying this is stating the obvious. Are you then trying to say: “Right at this very moment we can push a bit further with more data/computation given what we have for this specific task”, well, then again this is nothing particularly insightful to bring to the table as surely this occurs at least once a month for a given task/model category. This is not a matter of clarity, it is a matter of form – drop the hyperbole and speak specifics rather than trying to appease both sides of a fictitious divide.
Cybiote 8y ago