This essay sort of waves a hand at the sub-symbolic roots of knowledge that lie beneath text and that babies spend several years mastering before they are ever

This essay sort of waves a hand at the sub-symbolic roots of knowledge that lie beneath text and that babies spend several years mastering before they are ever exposed to text. IMHO the proper measure of that latent knowledge is qualitative, not quantitative. It's the tacit 'grounded knowledge' of the world that's present in humans that has the potential to fully fill in LLMs' causal blank in their text-based superficial info. This kind of knowledge is threadbare in today's LLMs, but essential to form a basis for further self-education in any intelligent agent. I know STaR and RLHF have been suggested as synthetic means to achieve that experimental end, but I'm not sure they're sufficient to connect the dots between LLMs' high-level book learning and human babies' low-level experiment-based intuition for cause and effect. But adding yet more text data is surely NOT the way to span that chasm.