Thats because the one unstated rule is clear: Don't lead the owners of the house to believe you should no longer be invited. There, it doesn’t matter if its no

Thats because the one unstated rule is clear: Don't lead the owners of the house to believe you should no longer be invited. There, it doesn’t matter if its not a “violation”, you can get excluded if you’re just bad at making socially-appropriate eye-contact or if you’re a boring conversationalist or whatever. But some people (myself included) feel really uncomfortable with that model because they aren’t confident in their social skills.