Making larger and larger datasets works fine for text and images. The internet has so much of that we can scrape superhuman quantities of the stuff and shove it

Making larger and larger datasets works fine for text and images. The internet has so much of that we can scrape superhuman quantities of the stuff and shove it in to a next token predictor and the result is pretty good. But what about for tasks where datasets don’t really exist? I do a lot of PCB design and it’s extremely time consuming. But it’s a niche field compared to text and images. No dataset exists that says “these were the engineering requirements of this PCB and this is the result and by the way the board actually worked”. So how will we train AI systems to replace a human doing PCB design? It’s probably going to need to learn PCB design from first principles (along with massive help from large transformers when possible, like collections of chip datasheets). Even then, understanding PDF datasheets is something these big companies haven’t really pulled off yet, though I suspect in 5 years that will change. But my point is that there must be loads of tasks, even on the computer, for which suitable datasets don’t exist and it would be infeasible to create them. Another big thing I do is machine design and again it’s not about designing one mechanical part it’s about pulling in the right parts from all over the world and assuming certain manufacturing processes, and then knowing those processes and then designing all the parts and the assembly. There’s so many different pieces of knowledge in there that are not captured on text or images on the web and would be hard to encode in to datasets. At some point we’re going to need machine systems that learn the way people do, and that’s going to take a long time to figure out. That’s what LeCun is saying.