With regard to model selection, one thing I learned a long time ago that provides powerful intuitive guidance on which model to use is the question: "How could

With regard to model selection, one thing I learned a long time ago that provides powerful intuitive guidance on which model to use is the question: "How could this be compressed further?" There are some deep connections between data compression and generalized learning, both at the statistical level and even lower at the algorithmic level (see Solomonoff induction). For a specific example at the statistical level, suppose you fit a linear trendline to some data points using OLS. Now compute the standard deviation of the residual terms for each data point, and using the CDF of the normal distribution, for each residual, map its value into the interval [0, 1]. Sum together the logarithms of the trendline coefficients, the standard deviation, and the normalized residuals. This value is approximately proportional to "sizeof(data | model) + sizeof(model)". It represents how well you compressed the data using the OLS model. But now suppose you plot the distribution of the residuals and find out that they do not in fact resemble a Gaussian distribution. This is a problem because our model assumed the error terms were distributed normally, and since this is not the case, our compression is suboptimal. So you back out some function f that closely maps between the uniform distribution on [0, 1] and the distribution that the residuals form and use this f to define a new model: yᵢ = m*xᵢ + b + εᵢ, with εᵢ distributed according to f(x;Θ), Θ being a parameter vector. When you sum the logarithms again, you will find that the new total is smaller than the original total obtained using OLS. The new trend line coefficients will slightly mess up the residual distribution again, so iterate on this process until you've converged on stable values for m, b, and Θ. At the algorithmic level, the recommendation to use compression as a model selection guide applies even to LLMs, but it's a bit harder to use in practice because "sizeof(data | model) + sizeof(model)" isn't the entire story here. Suppose you had a "perfect" language model. In this case, you would achieve minimization of K(data | training data), where K is Kolmogorov complexity. In practice, what is being minimized with each new LLM version is "sizeof(data | LLM model) + sizeof(training data | LLM model) + sizeof(LLM model)". You can assume "sizeof(LLM model)" is the smallest Turing machine equivalent to the LLM program.