You don't have a right to "value". The exact same argument works against you. Preventing me from building housing hurts the poor! It makes their cost of living

You don't have a right to "value". The exact same argument works against you. Preventing me from building housing hurts the poor! It makes their cost of living increase. And surely they suffer far more than rich property owners. All economic transactions hurt someone's value, but they create more value than they destroy. It's a net good.