4 ms·
The important distinction that I'm not sure some of the commenters here are picking up on is that this test is not being carried out on data in the "chat" histo
by montebicyclelo 3y ago
The important distinction that I'm not sure some of the commenters here are picking up on is that this test is not being carried out on data in the "chat" history (i.e. context window), it's on data in the training history.
E.g. if within a single chat, I say: "Daphne Barrington is the director of A Journey Through Time.", (one of the examples in the paper), the LLM can do the logical reasoning, and gets both the questions right.
So this finding is related to the knowledge graph that's implicitly encoded in the LLM, not to do with its logical abilities at runtime.
- yladiz 3y agoIn the simple case your example might work, but we know that the LLMs are not able to make general logical inferences at runtime with both strong knowledge of a subject and runtime data, in the case of an LLM playing chess. Regardless of its actual ability, LLMs state purely impossible and/or illegal moves and states moves are not legal when they are, which means it’s unable to understand (learn) the basic rules of chess.