2 ms·
Short: The short answer is that it is hard to compare models so to make it easier we compare parameters. Part of the answer of why we do it is because it also
by godelski 3y ago
Short:
The short answer is that it is hard to compare models so to make it easier we compare parameters. Part of the answer of why we do it is because it also helps show scaling. (As far as I'm aware) The parameters __are not__ optimal, and we have no idea what actually that would mean.
Longer:
The longer answer is that comparing models is really fucking hard and how we tend to do it in the real world is not that great. You have to think of papers and experiments as proxies, but proxies for what? There's so many things that you need to compare a model on and it is actually really difficult to convey. Are you just trying to get the best performance? Are you trying to demonstrate a better architecture? Are you increasing speed? Are you increasing generalization (note the difference from performance)? And so on. Then we need to get into the actual metrics. What do the metrics mean? What are their limitations? What do they actually convey? These parts are unfortunately not asked as much but note that all metrics are models too (everything you touch is "a model"), and remember that "all models are wrong." It's important to remember that there are hundreds or thousands of metrics out there and they all have different biases and limitations, with no single metric being able to properly convey how good a model is at any task you choose. There is no "best language model" metric, nor are there even more specific "best at writing leet code style problems in python" metrics (though we'd be better at capturing that than the former question). Metrics are only guides and you must be truly aware of their limitations to properly evaluate (especially when we talk about high dimensions). This is why I rant about math in ML: You don't need to know math to make a good model, but you do need to know math to know why a model is wrong.
Parameters (along with GMACs, which is dominating the FLOPs camp. Similarly inference speeds have become common place) only started to be included as common practice in the last few years and still not in every subject (tends to be around the transformer projects, both language and vision). As a quick example of why we want them, check out DDP vs iDDPM. You wouldn't know that the models are about 60% different in parameter size when comparing (Table 3). In fact, you're going to have a hard time noticing the difference unless you read both very carefully as they're both one liners (or just load the models. fucking tensorflow 1.15...). Does it seem fair to compare these two models? Obviously it depends, right? Is it fair to compare LLaMA 2 70B to LLaMA 2 7B? It both is and isn't. It entirely depends on what your needs are, but these are quite difficult to accurately capture. If my needs are to run on device in a mobile phone 7B probably wins hands down, but this would flip if I'm running on a server. The thing is that we just need to be clear about our goals, right? The more specific we can get about goals, the more specific we can get around comparing.
But there's also weird effects that the metrics (remember, these are models too. Ask models of what) we use aren't entirely capturing. You may notice that some models have "better scores" but don't seem to work as well in real world use, right? Those are limitations of the metrics. While a better negative log likelihood/entropy score correlates well with being a high performing language model, it does not __mean__ a high performing language model. Entropy is a capture of information (but make sure not to conflate with the vernacular definition). These models are also very specifically difficult to evaluate given that they are trained and tested on different datasets (I absolutely rage here because non-hacking can't be verified) as well as the alignment done post process. This all gets incredibly complex and the honest truth here is that I don't think there is enough discussion around the topic of what a clusterfuck it is to compare models. Hell, it is hard to even compare more simple models doing more simple tasks like even just classifying MNIST numbers. Much harder than you might think. And don't get me started on out of distribution, generalization, and/or alignment.
I would just say: if you're a layman, just watch and judge by how useful the tools are to you as a user -- be excited about the progress but don't let people sell you snake oil; if you're a researcher, why the fuck are we getting more lazy in evaluating works as the complexity of evaluation is exponentially increasing -- seriously, what the fuck is wrong with us?