Smith is speaking about the core business of owning and letting use of land (or some other finite resource) itself. The improvements that one might speculative

Smith is speaking about the core business of owning and letting use of land (or some other finite resource) itself. The improvements that one might speculatively build on the land and then need to maintain or alter over time is actually a different, secondary business which can be seen as more labor-based, as you note.