Having read the paper: They evaluated their model by producing a test set with videos spliced so as to compose physically "impossible" sequences (that's the cl

Having read the paper: They evaluated their model by producing a test set with videos spliced so as to compose physically "impossible" sequences (that's the claim anyway - who knows how "impossible" those sequences are). Then they measured the trained network's prediction error on the spliced videos and they called that "surprise", implying that it is somehow similar to the "surprise" shown by human babies when they observe physically impossible sequences (to clarify, in the psych literature this "surprise" is measured as a function of the time spent looking at a scene). Then they measured "accuracy" as the difference in "surprise" between possible and impossible videos. There are two problems with this. First, it's the old switcheroo: it just relabels prediction error as "surprise". Yeah, "surprise" is interpreted, in a Bayesian setting, as the difference between an expected event and an actually occuring event but here we're not in Bayesian world. There's just a classifier failing to classify correctly. By the same token, any classifier's classification error ever can be interpreted as "surprise", even though it's just failure to acquire the target concept. Second, this measure of "surprise" has nothing to do with the measure of surprise used in experiments with baby humans, where "surprise" is a function of the time the baby spends looking at a scene. For a baby human it makes sense to intrepret longer time looking as a double-take: "what the hell just happened"? But for a neural net? How can a neural net do a double-take? First of all the time spent "looking" at any image is entirely controlled by the experimenters. This is just an apples-to-oranges comparison and the result is a lemon. Third, their "flat" networks trained without hand-crafted object segmentation show lower "surprise" but since that's just the sum of squared errors, it only means that the flat networks have learned classifiers that perform better on the test set than the proposed approach. When "accuracy" is compared, the flat networks score lower but that should be expected: if "accuracy" is measured as the difference in classification error, the model that is better at classifying across all tasks should have lower "accuracy". But that's because it has higher classification accuracy. (yeah, that's an off-by-one error, well spotted). Then, to test whether their system generalises to unseen objects and events, they tested it on a separate dataset, called ADEPT, and found that it scores very well. The only problems is that they measured "accuracy" again as the difference in mean squared errors between "possible" and "impossible" scenes. All this is measuring is the _lack_ of generalisation from training to test images. I'm sorry to be so negative, again, but all I see here is playing with language and rechristening error as "surprise" and "accuracy". There is nothing that shows acquisition of "intuitive physics" like the paper keeps claiming.