That was my initial reaction as well but they validate on separate datasets from training which makes this unlikely. The performance despite degradation may be

That was my initial reaction as well but they validate on separate datasets from training which makes this unlikely. The performance despite degradation may be the same phenomenon that results in adversarial examples that are indistinguishable to human eyes, ie we know that neural nets are highly sensitive to visually imperceptible differences.