That’s absolutely false. 1) statistical intervals are mathematical artifacts of our techniques that describe the samples observed and trained on. They aren’t g

That’s absolutely false. 1) statistical intervals are mathematical artifacts of our techniques that describe the samples observed and trained on. They aren’t ground truth observations of the underlying process or populations. We put too much weight in them. 2) you can absolutely observe precision and recall from online performance and compare that directly against human performance on known labeled data. From that you can determine which has the better error rates. That is entirely sufficient for almost all practical use cases. 3) obviously it would be better if we could derive confidence of a classification, but given the fact LLMs aren’t directly reasoning or optimizing the statistical properties of some mathematical problem they will never have the same character as say regressions or other statistical techniques that are some form of mathematical optimizer. They’re just solving the problem in fundamentally different ways. 4) it’s not clear to me statistical optimization is a universally superior technique. The reality is many problems are better solved with an abductive reasoning technique like LLMs exhibit, and humans absolutely use when classifying. There are lots of awesome features such as the ability to inspect residuals and confidence intervals, and they’re generally computationally cheap. But for all that their absolute utility in the real world is fairly limited, especially when considering complex non linear tasks with huge latent spaces that are unobservable.