That depends on context and location. Where I'm from, wearing your football's team shirt is nothing to be scoffed at. That is, except by your rivals, but they'

That depends on context and location. Where I'm from, wearing your football's team shirt is nothing to be scoffed at. That is, except by your rivals, but they're not taking you any less seriously - just not liking your choice of team. And yes, it's very normal to wear that to work, especially (but not exclusively) after a big match where your team won or played particularly well, or at least bravely.