I always understood the version with no apostrophe as a general imperative statement, with 'hats' being plural - not an abbreviation of 'my hat's off' but of 'w

I always understood the version with no apostrophe as a general imperative statement, with 'hats' being plural - not an abbreviation of 'my hat's off' but of 'we should take our hats off'. Either makes sense.