It's an idiom so the meaning is inherently somewhat divorced from the literal meaning of the words. In this case nearly opposite of it but that's not super unus

It's an idiom so the meaning is inherently somewhat divorced from the literal meaning of the words. In this case nearly opposite of it but that's not super unusual itself. It also has an equivalent where the component words are flipped, but the idiom's meaning is the same. Also not extremely rare. If someone says this with the "wrong" meaning, and a listener understands what they meant without thinking about it or considering alternatives. And you crash in saying "you're wrong!"... who is wrong there?