When a model is trained you end up with nothing more than a bunch of weights. These weights are used to predict the next token in a sequence. LLM models do not

When a model is trained you end up with nothing more than a bunch of weights. These weights are used to predict the next token in a sequence. LLM models do not have an external memory. LLMs only retain enough knowledge to make it into the next round of training. The astonishing thing here is that even pretty small models now know so much that people assume you can ask knowledge questions about any subject under the sun and get a factual answer. Absurd of course. Logically impossible -- it follows trivially from the size of the model used. The only thing the author has proven with his little test is his own lack of scientific curiosity. For any question that requires research (or deep expertise in a specific field) you need to use either a research model (that can reason and look things up in external knowledge bases) or you need a model that is trained on the kind of questions that you want to ask it so that it retains that data.