Yet, if an ML model can act as a reasonably accurate classifier on that data, the presence of an internal structure of that data is proven. In other words, ther

Yet, if an ML model can act as a reasonably accurate classifier on that data, the presence of an internal structure of that data is proven. In other words, there exists a transformation to a space in which the data set separates along lines similar to what is desired. Yet, without any kind understanding of that transformation, we cannot reason about its properties in a meaningful way. This is the downfall of the current generation of succesful ML models. We may not need an exact understanding of a training result. An approximate one may be enough, depending on the kind of insight that needs to be extracted. My pipedream vision is that ML models are some day just mere tools that help design simpler models with formally guaranteed properties.