4 ms·
The point about peeking at the holdout set is fine, I guess. But then the slogan should be "models without a truly blinded holdout set are statistical fiction."
by spekcular 5y ago
The point about peeking at the holdout set is fine, I guess. But then the slogan should be "models without a truly blinded holdout set are statistical fiction." Prospectivity doesn't do any work above and beyond blinding there.
I already addressed the point about distribution shift above. You need to assume that the past reflects the future at some point, otherwise you can't justify deploying the model even after the prospective test.