Yes, exactly. But I'd extend this one part: > Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tw

Yes, exactly. But I'd extend this one part: > Almost everyone is going to either peek at the data, massage the data, or try the validation and when it fails tweak the model and try again, or any number of shenanigans that no one admits happen - but happen all the time. While "peeking" at the data is certainly something that happens all the time, usually what happens for even the most rigorous of model-makers is that your model fails "out of domain" -- the past doesn't accurately reflect the future. Even if you get everything else right, it's incredibly common to fool yourself into believing you're capturing "reality", when in fact you're just making a good fit to past random fluctuations.