There's no absolute "should" for what you're talking about, but the argument most economists would make is that most other tax schemes other than land value tax

There's no absolute "should" for what you're talking about, but the argument most economists would make is that most other tax schemes other than land value taxes are economically inefficient. One way this can manifest is deadweight loss. This means that we misallocate resources. For instance, consider a man who buys a lot of land next to city hall. He does nothing with it for years as the city grows up around him. His lot is now worth a lot of money and he pays very little for it yearly. One way of looking at this is in terms of how much it costs to provide him services: nothing, so his property tax is net positive. Another way is that there is a net loss to society because his property is not put to use in an economically efficient way. Economically efficient does not necessarily mean developed, by the way. Golden Gate Park may be an economically efficient allocation of resources, though it is not used for built up space.