One thing I don't get with libertarianism is why this, second paragraph of the book, doesn't apply both ways: "People who wish to live in a 'virtuous' society,

One thing I don't get with libertarianism is why this, second paragraph of the book, doesn't apply both ways: "People who wish to live in a 'virtuous' society, surrounded by others who share their ideas of virtue, would be free to set up their own communities and to contract with each other so as to prevent the 'sinful' from buying or renting within them. Those who wished to live communally could set up their own communes. But nobody would have a right to force his way of life upon his neighbor."