I have a general unease with the argument from datasets. It’s true for the current paradigm, but at some point, you can just plug an agent into a robot body and

I have a general unease with the argument from datasets. It’s true for the current paradigm, but at some point, you can just plug an agent into a robot body and have it generate and predict arbitrary cause/effect in the world. Or plug in the feed of internet data and have it predict stocks. The reason these don’t work now is that the gradients are way too sparse to learn from. But at a certain level of capability this will stop being true. The big Q is whether the existing gradient of “all written content” gets the loss low enough for another information source to smoothly take over. More pithily, there are plenty of high-quality datasets if you are smart enough to generate OK predictions. The smarter you get the more you can learn from. There are plenty of obvious synthetic datasets that GPT-5 could be trained on, eg generated problems for all of the physics/maths it currently gets wrong. (I’d be surprised if this isn’t an angle they are working on.)