4 ms·
I can already answer you that question based on the "reversal curse" paper. LLMs trained on “A is B” fail to learn “B is A”. It will only work if you explicitl
by visarga 3y ago
I can already answer you that question based on the "reversal curse" paper. LLMs trained on “A is B” fail to learn “B is A”. It will only work if you explicitly generate text connecting B to A before training.
More generally, the organic training data is almost always incomplete. Samples of text leave many assumptions and inferences hidden. Like a math problem, you see the statement, but don't know the answer until you work it out. Or like a puzzle, you see the pieces but don't know the big picture until you fit those pieces together.
That is how training text samples are like unsolved enigmas, we train our models on undigested text. Often the pieces are spread over many training examples that almost aways appear separately, never together have the chance to draw a conclusion from them. Search is needed, augmenting training examples with supporting data.
Neural nets are smart at inference time but dumb at training time. They don't make those connections when they train. Instead, we need to draw those connections out by generating new text. We need to benefit from inference-time smarts before training. That means we need to use current LLMs to write the dataset of next LLMs.
All the best LLMs today used a big piece of synthetic data, including GPT-4. Datasets like Orca, Phi-1.5, ShareGPT, etc. It's also the best way to create small models (<10B) that actually work, you need very high quality, high diversity synthetic data.
- nielsole 3y agoI wonder how well it would work if llms would do "online learning". After every interaction they'd be asked to summarise new knowledge (assuming the input is trusted) and create multiple variations on it, then train on it. I wonder if such a model would improve or degrade over time.
- tomohelix 3y agoI don't get it. A is B doesn't always mean B is A.
- ngneer 3y agoCould you give an example?
- warkdarrior 3y agoCows are mammals, but mammals are not cows.
- sam_goody 3y ago"Cows are mammals" means: Array($mammals).includes($cows). It does not mean $cows === $mammals. "A is a B" Or "A's are B's" means that A is included in B. $a === $b ("A is B") implies that $b === $a (B is A), in all cases that I know of.
- ToValueFunfetti 3y agoWhat cases are there for $a === $b? I guess there's synonyms and rephrasing, but in general it's not a useful relationship in language
- wizzwizz4 3y agoBatman is Bruce Wayne.
- xcv123 3y agoFrom this paper: https://owainevans.github.io/reversal_curse.pdf https://owainevans.github.io/reversal_curse.pdf "In particular, suppose that a model’s training set contains sentences like “Olaf Scholz was the ninth Chancellor of Germany”, where the name “Olaf Scholz” precedes the description “the ninth Chancellor of Germany”. Then the model may learn to answer correctly to “Who was Olaf Scholz? [A: The ninth Chancellor of Germany]”. But it will fail to answer “Who was the ninth Chancellor of Germany?” and any other prompts where the description precedes the name."
- akomtu 3y agoThe operator are, which happens to look similar to an english word, has a meaning that depends on context. Another example is 5/2 means a simple fraction in one context, or a class of polytopes in another.
- PoignardAzur 3y agoGP meant that LLMs trained to learned "A is a B" won't necessarily be able to answer "give an example of B". Eg given the fact "Foo is the capital city of Bar", it might be able to complete "Foo is the capital city of _" but not "The capital city of Bar is _". (Which isn't necessarily a evidence against truth models for out-of-distribution facts, it could just be a matter of indexing.)
- xcv123 3y agoYes it can only learn to solve the problems that it was trained to solve. Not surprising really. Tautological.
- ngneer 3y agoRight, but that ignores the key piece, which is whether such systems can infer the equivalence relation denoted by "is". A is A. A is B implies B is A. A is B and B is C implies A is C. When a system sees that A is B, yet cannot infer that B is A, it exhibits an asymmetry, which is interesting. Whether this asymmetry exists in the underlying language is unclear.
- xcv123 3y ago> When a system sees that A is B, yet cannot infer that B is A So the issue is that they cannot infer that general rule due to a fundamental limitation of the transformer LLM architecture, not just a training data issue? I skimmed the paper and it seems to be the case.
- dr_kiszonka 3y ago> A is B implies B is A. I get the overall idea, but this statement isn't always true, right?
- xcv123 3y agoCorrect. Not in natural language. These are natural language systems, not logic systems. "The sky is blue" does not imply "blue is the sky".
- kybernetikos 3y agoIf I say to chat gpt Adrian Tompkins was the ninth mayor of the town of Wolverhillington. Who was the ninth mayor of Wolverhillington? Then it correctly responds The ninth mayor of the town of Wolverhillington was Adrian Tompkins. What am I doing wrong?
- famouswaffles 3y agoNothing. It's not an inference problem but a training one. If you train on the sequence, "Adrian Tompkins was the ninth mayor of the town of Wolverhillington" And later ask in inference, "Who was the ninth mayor of Wolverhillington?", It might not return the answer.