3 ms·
Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities). That's the sor
by fluidcruft 7mo ago
Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities). That's the sort of thing that would make me go dig to find where I fucked up the calculation.
- kqr 7mo agoWith one term it gets more robust in the face of excluding endpoints when constructing the jackknife train/test split, I think. But you're right, it does sound fishy.
- fluidcruft 7mo agoWhat the post is describing is just ANOVA. If removing a category improves the overall fit then fitting the two terms independently has the same optimal solution (with the two independent terms found to be identical). MSE never increases when adding a category. This is why you have to reach to things that penalize adding parameters to models when running model comparisons.
- kqr 7mo agoNo, the post is doing cross-validation to test predictive power directly. The error will not decompose as neatly then.
- fluidcruft 7mo agoWhy would they do that and where do you see evidence they did?
- kqr 7mo agoBecause it's a direct way to measure predictive power, and it says so: "We’ll use leave-one-out cross-validation"