Predicting the future. I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861 https://arxiv.org/abs

Predicting the future. I recently saw a paper getting LLM evaluation completely right in many respects: https://arxiv.org/abs//2402.00861 https://arxiv.org/abs//2402.00861 "Evaluating Large Language Models for Generalization and Robustness via Data Compression" They evaluate LLMs by seeing how well they compress text/news/science from the future, over time. In a sense, this measures how "surprised" the LLM is by future developments. This is, I believe, an important and strong task. A top researcher or economist will have a preternatural ability to "see where the puck is going" in a way that would be more surprising (perplexing) to a junior person or a layperson. Thus, things after the knowledge cutoff being more "expected" is a great sign of predictive generality.