The idea you're looking for is the paradox of tolerance, and a solution to it is that by engaging in intolerance, one has broken a social contract, and thus no

The idea you're looking for is the paradox of tolerance, and a solution to it is that by engaging in intolerance, one has broken a social contract, and thus no longer benefits from it, and others can be intolerant to them without violating this social contract.