5 ms·
The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?). From the first (of three) linked papers in the a
by karmakaze 5y ago
The number of parameters could be a vanity metric--like saying my CPU is 1000W (is that fast or inefficient?).
From the first (of three) linked papers in the article.
> Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world. In this paper, we present an analysis of Transformer-based language model performance across a wide range of model scales — from models with tens of millions of parameters up to a 280 billion parameter model called Gopher. These models are evaluated on 152 diverse tasks, achieving state-of-the-art performance across the majority. Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit. We provide a holistic analysis of the training dataset and model’s behaviour, covering the intersection of model scale with bias and toxicity. Finally we discuss the application of language models to AI safety and the
mitigation of downstream harms.
The researchers seem to know what they're doing and not doing.
- piyh 5y agohttps://youtu.be/ujMvnQpP528 https://youtu.be/ujMvnQpP528 Microsoft is saying that more parameters is inherently better. I'm sure there's reasonable limits that they need to be competently implemented.
- YetAnotherNick 5y agoI remember some talk in which the researcher quantitatively said that parameter count is the only important metric for transformers it seems. It doesn't matter if the model is taller or thicker or have more number of attention heads.
- ShamelessC 5y agoThat's correct. The original paper is "Scaling Laws for Transformers".
- minimaxir 5y agoThe number of parameters is absolutely not a vanity metric, but it is inefficient, and efficiency is a related-but-different area of research (e.g. distillation) that's more of a business problem than a research problem.
- YeGoblynQueenne 5y ago>> Gains from scale are largest in areas such as reading comprehension, fact-checking, and the identification of toxic language, but logical and mathematical reasoning see less benefit Left unsaid: it's much harder to learn shortcuts to fake performance in logic and reasoning and common arithmetic, than it is to do the same in "reading comprehension" and similar tasks for which there are no good benchmark datasets (most are basically multipl-choice tests that can be solved by a strong classifier without any need of anything like "comprehension") and for which there are no good metrics either (see BLEU score and friends).
- sailingparrot 5y agoAll large language models (GPT-2/3, GPT-Neo, Turing, Gopher), all use essentially the same architecture with some light variations, and the same datasets, again with some light variations on how filtering is done etc. As such there is no reason to expect them to be very different in term of efficiency, and it has been shown and well researched, that scaling the numbers of parameters directly correlates with improved model quality. So as long as you are comparing GPT style models to other GPT style models then parameter count is definitly not a vanity metric. This doesn't hold once you start comparing to e.g. mixture of experts model which were making the headline recently with trillion of parameters claims. In MoE models, parameter counts is pretty much a useless metric.