Don't think in terms of memorisation, but compression. These models are trained to compress knowledge and then to leverage it to produce useful text. They're no

Don't think in terms of memorisation, but compression. These models are trained to compress knowledge and then to leverage it to produce useful text. They're not remembering it, or storing it in a traditional sense that computers might. It's similar to the loose way humans remember little facts and things that we've seen or heard a lot. For example, you can ask ChatGPT about obscure Wikipedia content – obscure things that it's almost certainly read. You'll find it doesn't have any recollection of the information. Larger models should enable LLMs to draw on more the information they're trained on, and in more depth.