3 ms·
> Because the data are finite, the data could admit many close-to-optimal models that predict differently from each other: a large Rashomon set. Hmm. But the
by comex 7y ago
> Because the data are finite, the data could admit many close-to-optimal models that predict differently from each other: a large Rashomon set.
Hmm. But the ultimate goal isn’t actually to find a model that makes good predictions on the training data. It’s to find a model that makes good predictions on data that‘s ingested in the future when the algorithm is put to actual use. But the set of possible future data is infinite! (Or at least exponentially large, depending on whether the input fields have finite precision.)
- gmueckl 7y agoYet, if an ML model can act as a reasonably accurate classifier on that data, the presence of an internal structure of that data is proven. In other words, there exists a transformation to a space in which the data set separates along lines similar to what is desired. Yet, without any kind understanding of that transformation, we cannot reason about its properties in a meaningful way. This is the downfall of the current generation of succesful ML models. We may not need an exact understanding of a training result. An approximate one may be enough, depending on the kind of insight that needs to be extracted. My pipedream vision is that ML models are some day just mere tools that help design simpler models with formally guaranteed properties.
- jmmcd 7y agoThis is a different issue. We already make some assumptions about the relationship between training data and future data, even when not worried about interpretability, and these assumptions are enough for this paper's argument too.