6 ms·
I just submitted an article about a paper by Deepmind whose main conclusion is that "data, not size, is the currently active constraint on language modeling per
by satvikpendem 4y ago
I just submitted an article about a paper by Deepmind whose main conclusion is that "data, not size, is the currently active constraint on language modeling performance" [0]. This means that even if we have bigger models, with billions and trillions of parameters, they are unlikely to be better than our current ones, because our amount of data is the bottleneck.
TFA though also reminds me of the phenomenon in mathematical proofs where some long winded proof eventually comes up but then over time it becomes simplified as more mathematicians try to optimize it, in much the same way as programmers with technical debt ("make it work, make it right, make it fast"), such as with the four color theorem that was until now computer assisted but it seems there is a non computer assisted proof out [1].
I wonder if the problem in TFA could itself be solved by machine learning, where models would create, train, and test other models, changing them along the way, similar to genetic programming but with "artificial selection" and not "natural selection" so to speak.
[0] https://news.ycombinator.com/item?id=34098087 https://news.ycombinator.com/item?id=34098087
[1] https://news.ycombinator.com/item?id=34082022 https://news.ycombinator.com/item?id=34082022
- hooande 4y agotwo thoughts People love to say "it's early" and "it will improve" about ChatGPT. but amount of training data IS the dominant factor in determining the quality of the output, usually in logarithmic terms. it's already trained on the entire internet. it's hard to see how they'll be able to significantly increase that. And having models build models is drastically overrated. again, the accuracy/quality improvements are largely driven by the scale and diversity of the dataset. that's like 90% of the solution to any ml problem. choosing the right model and parameters is often a minor relative improvement
- tlarkworthy 4y agoAll the books. I think books might be better
- recuter 4y agoThere's only about 100m books (in English). About the same volume of text as the web all total.
- MonkeyClub 4y agoGenerally a book has deeper thinking than a webpage, though, I think that’s the crux of the GP’s clarification.
- disgruntledphd2 4y agoAnd the distribution of text will be very different for all books vs the internet.
- albert_e 4y agoYes. And (with scary privacy implications) maybe the next frontier is capturing all spoken language uttered by people in real-time and streaming it into a model that is being updated in near-real-time. And finally brain implants extracting unspoken thoughts and neural activity and combining it all. Extend it to non human life forms as well. (universal consciousness?)
- akomtu 4y agoIt's the future where everyone is a universal captcha solver for the needs of the Great AI. No freedom even in your thoughts.
- deleted 4y ago[deleted]
- theGnuMe 4y ago> it's hard to see how they'll be able to significantly increase that. With feedback.
- DenisM 4y agoYes. Which is why ChatGPT is open to public.
- not2b 4y agoPerhaps some sort of adversarial network approach could work better; models that learn to generate text and other models that try to distinguish AIs from humans, competing against each other. Also, children learning language benefit from constant feedback from people who have their best interest at heart ... that last part is important because of episodes like Microsoft's Tay where 4chan folks thought it would be fun to turn the chatbot into a fascist.
- ronsor 4y agoI think OpenAI has already published some research showing humans preferred smaller/fewer parameter models that were "better trained" through the use of human feedback. If there were a model that could adequately replace the role of the human, then that approach would probably work well.
- charcircuit 4y agoPassing the Turing test isn't the goal. You can have a useful model that isn't human like and can have a useless model that you can't tell isn't a human.
- not2b 4y agoCertainly. Approaches that have done better on the Turing test have used various tricks to avoid their lack of understanding, like playing a paranoid person. But some of the best chatbots give themselves away by getting stuck in a loop or demonstrating lack of basic intuition about the world that a three year old would have. Those things perhaps could be caught.
- geysersam 4y agoThe data is the bottleneck for the current generation of models. Better models/training strategies could very well change that in the next couple of decades.
- dr_dshiv 4y agoWell, diffusion networks seemed to work because they increased the amount of data by training so many examples of added noise and it’s removal. Some similar approach might be possible with text, too. Or, i guess they will talk to itself, like alpha go played itself.
- rhacker 4y agoThe next phase has got to be more and more private data sets that don't exist on the internet: Homes with Alexa, Ok Google, etc.. Beyond that, linkages into a human brain.
- bilsbie 4y agoYet humans learn on way less language data.
- Enginerrrd 4y agoI disagree. On pure # of words, sure. But for humans, language is actually just a compressed version of reality perceived through multiple senses / prediction + observation cycles / model paradigms / scales / contexts/ social cues, etc. and we get full access to the entire thing. So a single sentence is wrapped in orders of magnitude more data. We also get multiple modes of interconnected feedback. How to describe this? Let me use an analogy. In poker, different properties a player has statistically take different amounts of data to reach convergence: Some become evident in 10s of hands, some take 100s of hands, and some take 1000s, and some even take 10,000s before you get over 90% confidence. ....And yet, if you let a good player see your behavior on just one single hand that goes to showdown, a good human player will be able to estimate your playing style, skill, and where your stats will converge to with remarkable accuracy. They get to see how you acted pre-flop, on the flop, turn, and river, with the rich context of position, pot-size, and what the other players were doing during those times, along with the stakes and location you're playing at, what you're wearing, how you move and handle your chips, etc. etc.
- numpad0 4y agoWe also eat. It feels to me that better food with divergent micronutrients has positive performance implications. Maybe I’m just schizophrenic, but to me it just feels that way.
- fshbbdssbbgdd 4y agoTry training the model with free-range, locally-sourced electrons.
- marcosdumay 4y agoIt's less data anyway. There's no way to add up the data a person senses and get into a volume any similar to what's on the internet. But it may be better data.
- wincy 4y agoCouldn’t we start a website that just has humans tag stuff for machine learning, and make that tagged data set open? Does such a thing exist? I’ve heard the issues with Stable Diffusion and others is that the LAION-5B dataset is kind of terrible quality.
- PoignardAzur 4y agoSurgeAI's business model is to crowdsource these kinds of datasets, some of which it has release in the open.
- nl 4y ago> I’ve heard the issues with Stable Diffusion and others is that the LAION-5B dataset is kind of terrible quality. This is mostly wrong. It's possible to get higher quality results in specific domains by fine-tuning on carefully annotated datasets. However these higher quality results wouldn't be possible without the vast pre-training on the huge dataset LAION-5B provides. > humans tag stuff for machine learning, and make that tagged data set open? Does such a thing exist? Yes, there are lots. The size of LAION-5B is the innovation here.
- tabtab 4y agoRe: similar to genetic programming A genetical algorithm was also what I was thinking of. One could devise some kind of symbolic (textual) way to represent a wiring/circuit diagram (graph) and evolve the most efficient "learner" using mutation and cross-breeding (e-sex). The earliest GA I read about used Lisp dicing. As far as "easiest" AI for humans to work with, "Factor Tables" may be a way: https://github.com/RowColz/AI https://github.com/RowColz/AI AI tuning then becomes more like accounting instead of a lab with Doc Brown. Factor Tables are much easier to analyze, debug, and modularize than neural nets.
- joaogui1 4y agoThere's been work combining GA's and Architecture search for neural networks, the main keyword to search is NeuroEvolution, with NEAT being one of the first "good" algorithms for that (though scaling it up is hard)
- albertzeyer 4y ago> "data, not size, is the currently active constraint on language modeling performance" This is a bit incomplete. It goes both ways. Right in the abstract, it says: "the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled" You are probably referring to the statement: "We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant." So GPT3 and co would perform better with more data, but only up to a certain point. At that point, you would also need to scale up the model size again to get better performance. But also, GPT3 and co would perform better when they would be scaled down a bit, when you keep the same training data. That is actually what the Chinchilla paper does. Their Chinchilla model is smaller than GPT3 and others, trained with same amount of data (as Gopher), and performs better. I don't really see a problem in scaling up models even more. This paper just says that you should also scale up the training tokens equally. As far as I remember, this paper does not quite address whether you can use the same training data twice. I think all statements assume that every training token is only used once.
- joaogui1 4y agoChinchilla is trained with substantially more tokens then Gopher, 1.4T Vs 300B
- jacquesm 4y agoAnd you can bet that any file timestamped prior to GPT being released will carry extra weight in the future.
- jacquesm 4y agoI'm reminded of the way computer chess has evolved. It wasn't all that long ago that you needed rooms full of all kinds of special purpose power hungry hardware and then suddenly all that was gone and the strongest programs in the world fit in your pocket on a device using as much power as a small lightbulb. An extra level of understanding of the problem domain achieved through 'slow' methods can be an enormous difference in terms of practical applications.