What I get from this is that neural networks really have no idea what they are talking about (which of course is no news). How could they? Is a feature more imp

What I get from this is that neural networks really have no idea what they are talking about (which of course is no news). How could they? Is a feature more important or relevant than another just because of its frequency? People look at the output of neural networks and when they work they say - of course they do. And when they don't they rationalize it with "your model was wrong from the start". (This is similar to how Alpha-Zero "discovered" the optimal chess openings. Of course it did, it was bound to find them by the rules of chess).