IDK, we don't seem lacking in metrics. For measuring the quality of LLMs at the core task of language modeling (which then gets applied to other tasks), the cla

IDK, we don't seem lacking in metrics. For measuring the quality of LLMs at the core task of language modeling (which then gets applied to other tasks), the classic information theory metric of perplexity works; for evaluating how well they transfer to other tasks, we can (and do) use metrics of those specific tasks, like the tasks in SuperGLUE dataset. All reasonable LLM papers do evaluation on these metrics, and they don't really "make the metric the target", you really can't do much in the large model pretraining to target SuperGLUE specifically.