This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this

This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2] In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a way that produces novel emergent abilities not seen in other models. [3] This means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing. The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results. 1. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 2. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/213#64e8a3267ef2c72cbce22126 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 3. https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 4. https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 5. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644