The non-expansionists don't need to fight all possible expansionists. They only need to fight the ones that come around. If B can't beat A, A will soon encounte

The non-expansionists don't need to fight all possible expansionists. They only need to fight the ones that come around. If B can't beat A, A will soon encounter C, and then D and E, with a fresh roll of the dice each time. The bigger they get, the more adversaries they cross. We are not predicting the future, but trying to account for an observation. For the scenario to remain on the list of reasons why we haven't been visited, it suffices that it is possible. There are of course lots of other possible reasons, among them that there is nobody to visit.