I'm guessing it's more likely because it either makes you look older or just makes you stand out more. People feel the need to nod or otherwise acknowledge you

I'm guessing it's more likely because it either makes you look older or just makes you stand out more. People feel the need to nod or otherwise acknowledge you when you make eye contact and that probably happens more often when you are wearing an odd hat.