Yes exactly, they just barely work for the examples shown them. It is caused by the training set, the network structure, and the metric which are all intertwine

Yes exactly, they just barely work for the examples shown them. It is caused by the training set, the network structure, and the metric which are all intertwined. The design of the network is a direct consequence of the dataset+metric because researchers focus on accuracy scores against the test data. Given a better dataset and metric, researchers will solve it with a better network design. I guarantee it.