Working on nlme models for work these days - it does become a bit of a headache when asking “how much better” the model with -2LL=8000 is from the model with -2

Working on nlme models for work these days - it does become a bit of a headache when asking “how much better” the model with -2LL=8000 is from the model with -2LL=7995 obviously one is better “more likely given the data” but what if the better one used 2 more parameters and is hence more complex and might be overfitting the dataset? Well then there are all these “heuristics” to look at, AIC, BIC, some sort of trick with a chi^2 distribution function- these are all just ways to penalize the objective function based on the # of parameters but it’s somewhat debatable which one to apply when and I have read that some parameter estimation softwares don’t even compute these values in the exact same way - I am not a statistician by training I just apply “industry standard practices” in as reasonably intuitive a way as I can but my impression has always been that if you wander far enough into the weeds you’ll find that stats often becomes a debate between many approximations with different sorts of tradeoffs and much of this is smoothed over by the fancy scientific software packages that get used by non-statistics-researchers one of the most frustrating parts of my job is reproducing SAS output (an extensively used statistics product) using free R language tools since a SAS license costs more than some sports cars… But what is SAS actually doing? And it’s never just taking a mean or pooling variance in the standard way you’d read about in an intermediate stats textbook it’s always doing some slight adjustment based on this or that approximation or heuristic This tangent may have been unrelated or irrelevant but I’ve long concluded that in practice statistics is far less solved than people might expect if they’ve never had to reproduce any of the numbers given to them by statistical analysis