Just a note: Logistic regression on raw pixels already gets you 88% accuracy. Nearest-neighbors on raw pixels gets you 95% accuracy. You should not draw too mu

Just a note: Logistic regression on raw pixels already gets you 88% accuracy. Nearest-neighbors on raw pixels gets you 95% accuracy. You should not draw too much conclusion from a network with accuracy < 97%, because you probably just have bad hyperparameters (except for conclusions about which hyperparameters you need to tune).