4 ms·

Putting aside whether how we want to define "science" there are models which attempt to represent the underlying system (e.g. generative grammars) and other "mo

Putting aside whether how we want to define "science" there are models which attempt to represent the underlying system (e.g. generative grammars) and other "models" which merely try to predict the input/output behavior of those systems (e.g. neural nets). The latter typically doesn't lend any understanding of the underlying system (e.g. we don't understand much about language from a deep neural net translation system), which isn't surprising because they are designed for prediction not for representing the internal behavior of the system. To use one of Chomsky's examples, if you built a deep neural net that could predict the weather with 100% accuracy then you have performed an amazing feat of engineering that is incredibly useful to the world. But you haven't necessarily learned anything about the weather.

dragonwriter 10y ago
The problem there seems to be not one with statistical models, but with the fact that extracting a parisomonious mathematical model from a neural net is nontrivial (presumably, you can extract a mathematical model from the internal weights, etc., but for even known simple relationships neural net models produced without an a priori prediction of the general shape of the relationship tend to be very complex.) Of course, were the model really 100% accurate, then converting it to a the most parsimonious perfectly accurate model would be simply a matter of reduction; the real problem is that real neural net models are not 100% accurate, and are often far more complex than more accurate models, and often aren't convenient to analyze to deduce the more-accurate and simpler model.
- arcanus 10y ago
  > extracting a parisomonious mathematical model from a neural net is nontrivial Has this ever been done? It strikes me as impossible. > Of course, were the model really 100% accurate, then converting it to a the most parsimonious perfectly accurate model would be simple a matter of reduction There are entire sub-fields of physics focused on reduced order models and phenomenalogical prediction. It is not 'simply a matter of reduction', it is non-trivial and the resulting models are almost always highly imperfect.
  - dragonwriter 10y ago
    > Has this ever been done? Probably not. > It strikes me as impossible. "Impossible" is probably, in principle, not quite right, but when I said "nontrivial", I really did mean "almost certainly impractical in virtually all real cases". > There are entire sub-fields of physics focused on reduced order models and phenomenalogical prediction. It is not 'simply a matter of reduction', it is non-trivial and the resulting models are almost always highly imperfect. "Simply" there meant that it was just reduction, not that reduction is necessarily simple. The point was that in real-world cases, reduction to an equivalent model isn't the only problem.
- foobarqux 10y ago
  The problem is not one of minimality, but of meaning and exposition. If you model a system using the smallest possible mathematical model you don't, from that act alone, understand how the system works.

dragonwriter 10y ago

The problem there seems to be not one with statistical models, but with the fact that extracting a parisomonious mathematical model from a neural net is nontrivial (presumably, you can extract a mathematical model from the internal weights, etc., but for even known simple relationships neural net models produced without an a priori prediction of the general shape of the relationship tend to be very complex.) Of course, were the model really 100% accurate, then converting it to a the most parsimonious perfectly accurate model would be simply a matter of reduction; the real problem is that real neural net models are not 100% accurate, and are often far more complex than more accurate models, and often aren't convenient to analyze to deduce the more-accurate and simpler model.

arcanus 10y ago
> extracting a parisomonious mathematical model from a neural net is nontrivial Has this ever been done? It strikes me as impossible. > Of course, were the model really 100% accurate, then converting it to a the most parsimonious perfectly accurate model would be simple a matter of reduction There are entire sub-fields of physics focused on reduced order models and phenomenalogical prediction. It is not 'simply a matter of reduction', it is non-trivial and the resulting models are almost always highly imperfect.
- dragonwriter 10y ago
  > Has this ever been done? Probably not. > It strikes me as impossible. "Impossible" is probably, in principle, not quite right, but when I said "nontrivial", I really did mean "almost certainly impractical in virtually all real cases". > There are entire sub-fields of physics focused on reduced order models and phenomenalogical prediction. It is not 'simply a matter of reduction', it is non-trivial and the resulting models are almost always highly imperfect. "Simply" there meant that it was just reduction, not that reduction is necessarily simple. The point was that in real-world cases, reduction to an equivalent model isn't the only problem.
foobarqux 10y ago
The problem is not one of minimality, but of meaning and exposition. If you model a system using the smallest possible mathematical model you don't, from that act alone, understand how the system works.