Your doubt about more data -> better semantic knowledge about the world is well-placed and I freely admit that at this stage it's mostly conjecture, although GP

Your doubt about more data -> better semantic knowledge about the world is well-placed and I freely admit that at this stage it's mostly conjecture, although GPT-3 does provide some evidence. However, my point is that once this hurdle is overcome, building agency on top isn't that much of a leap.