I see what you mean now. This plus the comment from user `argonaut` helped me realize what I was missing. Not all of generalization error is because of overfitt

I see what you mean now. This plus the comment from user `argonaut` helped me realize what I was missing. Not all of generalization error is because of overfitting, and even if none of it is because of overfitting, the idea you describe would still allow for the attack.