What happens if you feed it the Riemann Hypothesis or the Collatz Conjecture? It’s impressive that the model is clearly learning a lot of facts in training tha

What happens if you feed it the Riemann Hypothesis or the Collatz Conjecture? It’s impressive that the model is clearly learning a lot of facts in training that it just memorizes—like the variance of the first m even numbers, and remarkable to me that these models can generate correct output with high probability when they have no actual understanding of mathematics. But perhaps that’s just an indication that my own ideas are wrong—to wit, maybe math just isn’t that hard (in the sense that the sequence of correct outputs to a given prompt are a highly constrained pattern built around successive substitutions in common templates, rather than searching for a needle in a practically-infinite haystack).