I've had the best luck explaining this in terms of causal inference. We all know that it's really easy to screw up or invent connections between things when yo

I've had the best luck explaining this in terms of causal inference. We all know that it's really easy to screw up or invent connections between things when you use observational data instead of a randomized controlled trial. Observational data contains weird connections that you often can't tease apart, whether it's because there are important aspects of the mechanisms that were unrecorded (missing features), selection bias (missing data), or because regularization forces your hand in interesting ways. Generally for statisticians, this leads to bias in parameter space. For ML practicioners, it's more interesting/useful to regard this as bias in function space. There are useful lessons from the causal inference world too. There is a whole field of tools to try to get unbiased parameters/functions from observational data. There are even models that can guarantee unbiased parameters/functions under pretty reasonable assumptions and reasonable conditions. There is a rapidly developing field around learning unbiased (estimates of) functions from less "perfect" data.