There are however many ML systems that do not work, and many more that do not work as intended. Have you seen how many people have been kicked out of insert bi

There are however many ML systems that do not work, and many more that do not work as intended. Have you seen how many people have been kicked out of insert big corp by some AI decision? That's a problem that comes back to what is well known in statistics: Your model of the PD may be wrong, and the crucial question is - how can you find out? The ML answer is quite simple: I know my model is right, because it works. But this is very limited. Theoretically, you could enumerate all sorts of issues here. At the fundamental level, you will find that no model, no approach and no identification strategy is assumption free. "My model work" is a very simple identification strategy. It assumes, among other things, that the output of both PD in training and the estimated model is congruent with the latent distribution I'll find in reality. Such attempts at induction are, naturally, not as easy as the formulation suggests. You speak of reality, and I find that we are faced with model failures almost every day on HN. And, with all due respect to all ML researchers, there are a lot of teams with smaller experience and resources than Google, Apple and Amazon. I think there's resistance in the ML community to the recognition that somehow, at some point, one will have to deal with icky issues such as causal inference. On the other hand, I think this is precisely where the field moves forward. Until then, there's at least two costs we can readily identify: "Outliers" - people being somehow mishandled by the model - a collection of which is highlighted every day on HN, and second, changes (external or endogenous) to the PD, temporarily breaking models more or less. Again, it's subtle, but it all comes down to identification. I always tell the story of an engineer designing a robot to layer sausage slices. Well-defined PD - as you would say - worked great. Up until someone left the outside door open. Suddenly, the algorithm could not handle the problem anymore. What's the PD, and what changed? Good to think about such things every once in a while.