Because they aren't noise: they just seem like noise to humans. If you add in noise, then you have to train the network to disregard that noise. And the advers

Because they aren't noise: they just seem like noise to humans. If you add in noise, then you have to train the network to disregard that noise. And the adversarial input will then be features that occur above this noise floor you ignoring.