When verification is possible, the interpolation machine starts to extrapolate. There is a process that generates ideas/solutions, and a process that tests the

When verification is possible, the interpolation machine starts to extrapolate. There is a process that generates ideas/solutions, and a process that tests them. An artist and a critic, a scientist and a lab. Together they form the experimentation loop. Let's take the game of Go for example. Testing who won a game is trivial. AlphaGo managed to beat humans in a few days of self-training. In other words, those edges you speak about can be pushed with massive search and verification. There is no reason we can't do massive search + verification for math and code. This is a good way to create training data where it doesn't exist in sufficient quantity. Other things can be simulated with expensive computation, and then "distilled" into fast neural networks. Then we apply the neural net to fast-search solutions. In the end we need to verify some of them (thinking of weather simulations, new materials, new drugs, ...) Also reminded of the recent AlphaTensor who leveraged massive learning from verification to beat Strassen's algorithm who was state of the art for 50 years. The is no reason neural nets should remain purely interpolative if we can manufacture good training data by running computation or experiments. Ideas are cheap, verification matters. Generative outputs are worthless without verification.