4 ms·
> Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at
by 6gvONxR4sf7o 4y ago
> Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half its computational budget (i.e., saving ∼4.4 million TPUv4 hours).
Are you allowed to call your own work impressive in your abstract? Cool work, but that line is "transcendent."
Anyways, aren't scaling laws more like O(whatever) asymptotics? Like if you reduce your sorting algo from 6.4 n^2 seconds to 3.2 n^2, you don't say you "transcended the scaling laws," even though you sped it up a very significant amount. Am I misunderstanding?
- hobs 4y agoYes, that's day one stuff that the coefficient gets thrown away.
- p1esk 4y agoaren't scaling laws more like O(whatever) asymptotics? Not if scaling is linear.
- Nevermark 4y ago“Transcending a scaling law” and cutting a constant in half, shouldn’t be considered synonymous. Nice result regardless.
- deleted 4y ago[deleted]
- IanCal 4y agoI don't think that's the scaling they're talking about. I think they're talking about how the models performance scales with parameter size.
- deleted 4y ago[deleted]
- Dylan16807 4y agoI was going to say it could affect the scaling if the speedup changes with model size. But I checked the paper and it's about a 2x speedup on the 8B model too.
- airgapstopgap 4y agoThey talk specifically of scaling laws investigated in papers like OpenAI's "Scaling Laws for Neural Language Models" (2020)[1] and Deepmind's "An empirical analysis of compute-optimal large language model training" (2022)[2] which showed that, as far as we could see at the time, for any sensibly designed language model nuances of architecture do not matter nearly as much as the sheer scale of the model, amount of compute spent on training it, and volume of training dataset (or, indeed, that they do not matter at all, or even that attempts to add some cleverness via architecture ultimately handicap the model at production scale). These papers have introduced so-called "scaling laws" – predictable, robust relationships between scaling parameters and loss (which is meaningfully related to textual coherence and apparent "understanding" in zero-shot tasks), and for many have heralded the age of throwing compute at the problem without academic tinkering, and possibly the final stretch before human-level text processing. See also Rich Sutton's Bitter Lesson [3]. In this sense it is remarkable when the curve predicted from "scaling laws" gets broken through, transcended, since it suggests there still exists a better Pareto frontier for text prediction. I see how this can appear normal; but the thing is, scaling laws have been well-validated. We are no longer in the early era to expect to reap substantial returns from something as simple as changing training objective. 1. https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361 2. https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training https://www.deepmind.com/publications/an-empirical-analysis-... 3. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- 6gvONxR4sf7o 4y agoIt seems like every other paper in the scaling laws lit finds different scaling laws (if you consider the constant factors important).