I think that's mainly because the mailman might not notice that the mail in your box is not your mail being left over from when he dropped it off the day before

I think that's mainly because the mailman might not notice that the mail in your box is not your mail being left over from when he dropped it off the day before. The flag lets him know what's remaining in the box is to be delivered. He could read the addresses but they have very busy routes!