If you ask an LLM to complete a sentence like '[Insert name] stole the fruit (true/false):<answer>' An aligned LLM will be biased towards refusing to answer at

If you ask an LLM to complete a sentence like '[Insert name] stole the fruit (true/false):<answer>' An aligned LLM will be biased towards refusing to answer at all with something like: "I can't tell you because I don't know them." An "uncensored" LLM will very happily return <"true"> or <"false"> with a probability attached to each. Even OpenAI's GPT-3 does with a low enough temperature. _ Of course, LLM attention doesn't work like that. The tokens are just a bag of numbers: - The fact the name 'John' is mentioned in the Bible a lot affects the distribution when you ask if any John stole, because John is always [7554] - The fact that 'Olf' is part of Adolf and Adolf Hitler is mentioned in a lot of negative sentences will drag the distribution, because 'Olf' is always [4024] and Adolf is always [324, 4024] You could have asked something with no logical probability difference at all, like: - 'The store attendant's name was [name], did the child in Long Island drop his ball (true/false):' And unless you train the model to give you disclaimers it still follows the instruction faithfully and returns true/false with probabilities, demonstrating a deep regression in reasoning... That's why for models past a certain size, alignment increases performance: https://arxiv.org/abs/2204.05862 https://arxiv.org/abs/2204.05862.