4 ms·

So these guys trained "an AI" to identify hidden variables by looking at videos of physical phenomena, without prior knowledge of physics (except what was impli

So these guys trained "an AI" to identify hidden variables by looking at videos of physical phenomena, without prior knowledge of physics (except what was implicitly and obscurely encoded in "the AI's" neural net architecture, and its training set and hyperparameters, by the researchers). The "AI" came up with a slightly off number and the researchers struggle to understand why. They struggle because "the AI" is a black box that lacks explainability and cannot produce an explanatory model, only a predictive model [1]. The simplest conclusion is that their "AI" has not learned any useful models, and has only learned to accurately predict their test set, to which the authors had acces throughout the training of "the AI" [2]. The simplest explanation is that their model isn't very good at doing what they wanted it to do, but they opt to explain it as a scientific mystery, instead. Why? It seems the only reason to try and see a scientific mystery where a simple failure would suffice as an explanation is because the model is "an AI". There seems to be some kind of expectation that "an AI" must have some deeper understanding of physics than humans, even when we don't understand what it's doing, even whe it's not doing very well. In short, this seems to be based on very wrong assumptions and to be coming up with very wrong conclusions, but then again it makes for a great headline and so here we are, discussing it on HN. ____________________ [1] To clarify: a predictive model is one that can predict novel events. An explainable model is one that can explain how it made a prediction. An explanatory model is one that explains how the world works and why certain predictions are true, or not. Predictive and explainable models are useful, but most scientists aim to build explanatory models, because an explanatory model is necessarily also explainable and ultimately better at predictions, while a predictive model is not necessarily explainable or explanatory and an explainable model is not necessarily predictive or explanatory. Deep neural nets can only build predictive models, which are sometimes explainable, but they can't build explanatory models. That's because they can only identify correlations in data, but not explain those correlations with generalised theories, based on previous knowledge. To put it plainly, neural nets can't come up with scientific theories, but they can estimate probabilities of things happening. But scientists can and want to come up with scientific theories, not just predictions. [2] When the experimenter has access to the test data and can tune the learner's model until it scores highly on the test data, that leads to a model that overfits to the test data. Such a model is useless for prediction over unseen data (i.e. data not available to the researchers during training).

nootropicat 4y ago
>To clarify: a predictive model is one that can predict novel events. An explainable model is one that can explain how it made a prediction. Sorry but that distinction doesn't mean anything falsifiable. Did you mean an explanatory model is one that's easier to understand for a human? If so, that's certainly useful, but it may well turn out that the simplest working model of physics is incomprehensible.
- YeGoblynQueenne 4y ago
  No, I distinguish between "explainable" and "explanatory". To give an example, the theory of epicycles is a predictive model, that is also explainable, but it is not explanatory. Kepler's laws of planetary motion are an explanatory model that is also explainable and predictive. The theory of epicycles is predictive because it predicts the motions of the planets, as they are observed in the night sky. It is explainable because anyone can perform the necessary calculations and understand how the predicted motions are, well, predicted. The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated. So an "explanatory" model is a theory that explains why things happen they way they happen. A predictive model only predicts that some things will happen. An explainable model explains why it made a prediction, but it does not explain why this prediction should hold or with what frequency. Another way to see this is that an explanatory model explains past observations and predicts future observations, while a predictive model only predicts future observations and has no explanatory power, cannot explain why past phenomena were observed. An "explainable" model is just a model that people can understand. It's not more complicated than that.
  - nootropicat 4y ago
    >The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated. That's not a qualitative difference, it's a quantitative difference. Here you somewhat implicitly claim that models of lower Kolmogorov complexity for same, or better, accuracy are better, but how is that 'explanatory'? The term has no meaning. Ironically Kepler's laws are a better equivalent of epicycles, because they reduce to an imperfect approximation of Newtonian mechanics. Predicting actual orbits in a Newtonian Solar system of perfectly spherical objects with fully known matter distribution can only be done with an iterative solution. The actual, physical reality continuously diverges from all current models, and all we can try to achieve is to reduce that error. It can't ever be eliminated unless the universe is in fact a simulation, and whatever is controlling it decides to copy humans into the parent world, then gives them external access to the full underlying state.

nootropicat 4y ago

>To clarify: a predictive model is one that can predict novel events. An explainable model is one that can explain how it made a prediction. Sorry but that distinction doesn't mean anything falsifiable. Did you mean an explanatory model is one that's easier to understand for a human? If so, that's certainly useful, but it may well turn out that the simplest working model of physics is incomprehensible.

YeGoblynQueenne 4y ago
No, I distinguish between "explainable" and "explanatory". To give an example, the theory of epicycles is a predictive model, that is also explainable, but it is not explanatory. Kepler's laws of planetary motion are an explanatory model that is also explainable and predictive. The theory of epicycles is predictive because it predicts the motions of the planets, as they are observed in the night sky. It is explainable because anyone can perform the necessary calculations and understand how the predicted motions are, well, predicted. The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated. So an "explanatory" model is a theory that explains why things happen they way they happen. A predictive model only predicts that some things will happen. An explainable model explains why it made a prediction, but it does not explain why this prediction should hold or with what frequency. Another way to see this is that an explanatory model explains past observations and predicts future observations, while a predictive model only predicts future observations and has no explanatory power, cannot explain why past phenomena were observed. An "explainable" model is just a model that people can understand. It's not more complicated than that.
- nootropicat 4y ago
  >The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated. That's not a qualitative difference, it's a quantitative difference. Here you somewhat implicitly claim that models of lower Kolmogorov complexity for same, or better, accuracy are better, but how is that 'explanatory'? The term has no meaning. Ironically Kepler's laws are a better equivalent of epicycles, because they reduce to an imperfect approximation of Newtonian mechanics. Predicting actual orbits in a Newtonian Solar system of perfectly spherical objects with fully known matter distribution can only be done with an iterative solution. The actual, physical reality continuously diverges from all current models, and all we can try to achieve is to reduce that error. It can't ever be eliminated unless the universe is in fact a simulation, and whatever is controlling it decides to copy humans into the parent world, then gives them external access to the full underlying state.