The point of that argument is never "This is perfect." It's exactly as you say - "This is not bad enough to cause serious harm and therefore" (they're saying) "

The point of that argument is never "This is perfect." It's exactly as you say - "This is not bad enough to cause serious harm and therefore" (they're saying) "not bad enough to get worked up over it." "It survived evolution" = "It won't kill you" = "It's good enough" They're setting the bar a bit lower than optimum.