I think that's exactly the point OP was making. They are saying that even in a situation like Manhattan where nobody had any idea that the land would one day be

I think that's exactly the point OP was making. They are saying that even in a situation like Manhattan where nobody had any idea that the land would one day be worth a fortune, the average annual percentage increase in land value has not actually been that high. Certainly if you picked some random plot of land that's not part of a huge city, the numbers would be even smaller.