You're talking as if all economic activity involves building things. That's not the real world. The main question we should be asking about growth is "what happ

You're talking as if all economic activity involves building things. That's not the real world. The main question we should be asking about growth is "what happens to productivity gains?" Do they accrue to everyone evenly? Or do they disproportionately accrue to a small number of people? It turns out that over the last four decades, the answer has overwhelmingly been "accrue to a small number of people", and most people have seen almost nothing in terms of gains. A big part of the reason why is that there are entire industries (finance) devoted to that accrual. As a result, the continually-growing pie doesn't matter, because every time the pie grows larger, a few people keep taking larger and larger slices of it.