The fact that you can prune a model will not make it smarter, the wall still stands. I think what explains the wall is the fact that we can't scale organic data

The fact that you can prune a model will not make it smarter, the wall still stands. I think what explains the wall is the fact that we can't scale organic data exponentially, and we have already covered the most useful types. Going forward we will accumulate truly useful data at a linear growing rate. This fundamentally breaks the scaling game. If your model and compute expand exponentially but your training data only linearly, the efficiency won't be the same. Synthetic data might help us pad up the training sets, but the most promising avenue I think is to use user-LLM chat logs. Those logs contain real world grounding and human in the loop. Millions of humans doing novel tasks. But that only scales linearly with time, as well. No way around it - we only once had the whole internet for the first time in the training set. After that it's linear time.