That is to say, if all five of the users with whom you've tested your application say it's confusing, or it sucks somehow, it is diminishingly likely for that p

That is to say, if all five of the users with whom you've tested your application say it's confusing, or it sucks somehow, it is diminishingly likely for that population to be the outlier [0], and if only you had tested with a few tens or dozens — let alone thousands — more, you'd see the true pattern... Yes, statistically, it's possible for outliers to bunch like that. It's also, statistically, far less likely. [0] Assuming, for sake of argument, a nominally representative test group.