Maybe I'm missing something here, but if you say "Excuse me sir! You dropped your wallet" while holding up their wallet clear to see, people think "Oh no! A mug

Maybe I'm missing something here, but if you say "Excuse me sir! You dropped your wallet" while holding up their wallet clear to see, people think "Oh no! A mugger! And they must have already robbed me, let's run for it" ?