The main problem is the extrapolation of the “small world” we construct with mathematics to the “large world” of an application. Your model can be completely i

The main problem is the extrapolation of the “small world” we construct with mathematics to the “large world” of an application. Your model can be completely internally consistent and allow you to study things in the context of your assumptions perfectly, but that doesn’t mean it doesn’t have critical flaws that disqualify any learning about the system you’ve modeled. The lack of understanding about what implications a representational choice have for the kinds of relationships a mathematical model can describe, and the lack of understanding concerning what are and aren’t critical domain components to model that’s difficult. People often forget that there are two translation steps that have to take place. Domain -> Math -> Domain. That second one is too often implicit or assumed to follow, especially in ML.