It is not a voluntary interaction between property owners. It is a voluntary interaction between the factory owner and his customers, who are the paying parties

It is not a voluntary interaction between property owners. It is a voluntary interaction between the factory owner and his customers, who are the paying parties and therefore the only entities of any consequence on the free market. It's called an externality for a reason, being that the free market cannot account for it.