4 ms·
TL;DR: actually, they might. If I'm reading the visualization correctly, and not just projecting my own prior expectations, it seems to imply that a distinct an
by TeMPOraL 3y ago
TL;DR: actually, they might. If I'm reading the visualization correctly, and not just projecting my own prior expectations, it seems to imply that a distinct and general concept of "true" and "false" (or at least "known true" and "known false") emerges in the latent space.
It'll be interesting to explore how robust it is for statements crafted so that they surely did not occur in the training data, but their truth value could be indirectly inferred from information in the training data.
- visarga 3y agoI can already answer you that question based on the "reversal curse" paper. LLMs trained on “A is B” fail to learn “B is A”. It will only work if you explicitly generate text connecting B to A before training. More generally, the organic training data is almost always incomplete. Samples of text leave many assumptions and inferences hidden. Like a math problem, you see the statement, but don't know the answer until you work it out. Or like a puzzle, you see the pieces but don't know the big picture until you fit those pieces together. That is how training text samples are like unsolved enigmas, we train our models on undigested text. Often the pieces are spread over many training examples that almost aways appear separately, never together have the chance to draw a conclusion from them. Search is needed, augmenting training examples with supporting data. Neural nets are smart at inference time but dumb at training time. They don't make those connections when they train. Instead, we need to draw those connections out by generating new text. We need to benefit from inference-time smarts before training. That means we need to use current LLMs to write the dataset of next LLMs. All the best LLMs today used a big piece of synthetic data, including GPT-4. Datasets like Orca, Phi-1.5, ShareGPT, etc. It's also the best way to create small models (<10B) that actually work, you need very high quality, high diversity synthetic data.
- nielsole 3y agoI wonder how well it would work if llms would do "online learning". After every interaction they'd be asked to summarise new knowledge (assuming the input is trusted) and create multiple variations on it, then train on it. I wonder if such a model would improve or degrade over time.
- tomohelix 3y agoI don't get it. A is B doesn't always mean B is A.
- ngneer 3y agoCould you give an example?
- warkdarrior 3y agoCows are mammals, but mammals are not cows.
- sam_goody 3y ago"Cows are mammals" means: Array($mammals).includes($cows). It does not mean $cows === $mammals. "A is a B" Or "A's are B's" means that A is included in B. $a === $b ("A is B") implies that $b === $a (B is A), in all cases that I know of.
- ToValueFunfetti 3y agoWhat cases are there for $a === $b? I guess there's synonyms and rephrasing, but in general it's not a useful relationship in language
- wizzwizz4 3y agoBatman is Bruce Wayne.
- xcv123 3y agoFrom this paper: https://owainevans.github.io/reversal_curse.pdf https://owainevans.github.io/reversal_curse.pdf "In particular, suppose that a model’s training set contains sentences like “Olaf Scholz was the ninth Chancellor of Germany”, where the name “Olaf Scholz” precedes the description “the ninth Chancellor of Germany”. Then the model may learn to answer correctly to “Who was Olaf Scholz? [A: The ninth Chancellor of Germany]”. But it will fail to answer “Who was the ninth Chancellor of Germany?” and any other prompts where the description precedes the name."
- kybernetikos 3y agoIf I say to chat gpt Adrian Tompkins was the ninth mayor of the town of Wolverhillington. Who was the ninth mayor of Wolverhillington? Then it correctly responds The ninth mayor of the town of Wolverhillington was Adrian Tompkins. What am I doing wrong?
- famouswaffles 3y agoNothing. It's not an inference problem but a training one. If you train on the sequence, "Adrian Tompkins was the ninth mayor of the town of Wolverhillington" And later ask in inference, "Who was the ninth mayor of Wolverhillington?", It might not return the answer.