7 ms·
Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-
by ShamelessC 5y ago
Seems the link is down. Found a decent synopsis/discussion on lesswrong.
https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin...
> On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute.
> Following the new scaling laws that they propose for the optimal use of compute, DeepMind trains a new, 70-billion parameter model that outperforms much larger language models, including the 175-billion parameter GPT-3 and DeepMind's own 270-billion parameter "Gopher".
- eigenvalue 5y agoAccording to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger model on is “high school level math”! Very counterintuitive, and I am curious if there are any big takeaways lurking in that disparity.
- deleted 5y ago[deleted]
- adamsmith143 5y agoProbably right. Most people dump on these language models for this reason but it would be absurd for a HS student to have to re-derive the quadratic equation every time they worked on an Algebra problem so naturally you memorize it. Why should it be any different for a language model?
- eutectic 5y agoI never memorized the quadratic formula, and I did OK.
- whimsicalism 5y agoDid you go to school in the US in the last 2-3 decades?
- replygirl 5y agoOnce you start calculus they let you use a real calculator
- whimsicalism 5y agoThat may be true, but in the US there are typically math courses before calculus.
- replygirl 5y agoBut then we get a calculator.
- adamsmith143 5y agoMaybe we went to radically different schools but I certainly had to calculate by hand using the quadratic formula countless times where calculators were not allowed to be used. Anyway it distracts from the point so it's not relevant.
- whimsicalism 5y agoEven then, it is typically not a symbolic calculator so if your answer is a closed form function of variables, you're SOL with a TI-84.
- eutectic 5y agoNo, the UK. But completing the square only takes a minute.
- ShamelessC 5y agoGwern responded to a similar question in the comments section. (parent) > the fact that data and compute need to scale proportionally seems… like a big point in favor of NNs as memorizers/interpolators. (gwern) > Surely it's the opposite? The more bang you get out of each parameter, the less it looks like 'just' (whatever that means) memorization/interpolation. When you needed to increase parameters a lot, disproportionately, to cope with some more data, that does not speak well of abstractions or understanding. (If I can train a 1t model to get the same loss as what I thought was going to take a 100t model, why would I think that that 100t model must be memorizing/interpolating less?) Let's take your claim to its logical extreme: suppose we discovered tomorrow a scaling law that made parameters near-constant (log, let's say); would that not suggest that those parameters are super useful and it's doing an amazing job of learning the underlying algorithm and is not memorizing/interpolating?
- sillysaurusx 5y agoThis isn’t addressing their question. And Gwern’s goal here is to (incorrectly) try to get rid of the idea that models are just memorizing and interpolating, when in fact memorization and interpolation is what we all do, including models. He’s just bothered by the idea that people think of models as less than magic. On the other hand, https://twitter.com/model_mechanic/status/1512976881183645698?s=21 https://twitter.com/model_mechanic/status/151297688118364569... is admittedly pretty magical, even if the basis of that magic is memorization and interpolation.
- VirusNewbie 5y agoWhy do you say they just memorize and interpret? I can teach GPT-2 new things, including new objects and their physical properties and it does a good job with that. That also means it has definitely not just regurgitated a matching sentence back to me.
- replygirl 5y agowhen i see a new object for the first time, i MEMORIZE what i INTERPRET as its identifying traits, and ask someone who has already MEMORIZED what that object is to INTERPRET a concept with which i can associate those traits. the next time i encounter an object with those traits i can then recall the associations, then compose those trait-level interpretations into an interpretation of an object. at a fundamental level that's all this is, compositions of associated memorizations and interpretations, which map to compositions of sentence parts the machine can regurgitate back to you
- danuker 5y agoIt might just be by chance: the initial weights of one model could have been lucky in some areas, and unlucky in others. There's no way to tell other than training again, which is a costly proposition.
- eigenvalue 5y agoThat seems pretty unlikely to me actually. As the models and training data get much bigger, I think the initial weights become less important (at least assuming your random weights have certain desirable statistical properties, which they do by construction usually).
- VikingCoder 5y ago70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?
- sudosysgen 5y agoI'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.
- sillysaurusx 5y agoWelcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.
- native_samples 5y agoIs 100GB of parameters really that large? 128GB of RAM on a server class machine is not unusual. Seems such a model could fit entirely in RAM.
- andbberger 5y agoGPU memory is generally much smaller and more expensive
- kristjansson 5y agoTo elaborate on the sibling comment: main memory is much bigger, but CPUs are much, much slower. It would be a challenge to merely run a model like this on CPU, and totally infeasible to train one. So the challenge is to fit into the memory of a single GPU you can afford, coordinate multiple GPUs, or efficiently page from main memory into GPU.
- Delitio 5y agoIs there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, height,
- gyang 5y agoI think there remains an immense amount of such suboptimality still hanging from the tree, so to speak. For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows that even learning rate and initialization used by existing models are deeply wrong. By just picking them correctly (which involves some really beautiful mathematics), we can effectively double the model size of the GPT-3 6.7B model (to be comparable in quality to the 13B model across the suite of benchmark tasks). Large neural networks behave in a way we are only beginning to understand well just because each empirical probe of any such model is so much more expensive and time consuming than typical models. But principled theory here can have a lot of leverage by pointing out the right direction to look, as it did in our work. [1] http://arxiv.org/abs/2203.03466 http://arxiv.org/abs/2203.03466
- p1esk 5y agoWhat do you think about the concept of "critical batch size"? https://openai.com/blog/science-of-ai/ https://openai.com/blog/science-of-ai/
- gyang 5y agoI think the concept makes sense. The basic insight, that the right batch size depends on the difficulty and noisiness of a task, is already used by teams. For example, the PaLM paper from last week increased its batch size throughout training. But as far as I know, the more precise predictions of optimal batch size aren't used much, probably because it's expensive to measure accurately, or because the predictive equation isn't accurate enough to begin with. I wonder if we can "transfer" the optimal batch size from a smaller setting (smaller model or data) to the full setting, like in our paper. This would make it much more practical.