3 ms·
Language models can establish truthfulness in a formal logic system. There is a lot of literature that indicates this. Look at this for instance: https://openre
by computerex 4y ago
Language models can establish truthfulness in a formal logic system. There is a lot of literature that indicates this. Look at this for instance: https://openreview.net/forum?id=qFVVBzXxR2V https://openreview.net/forum?id=qFVVBzXxR2V
And you can test this quite easily with a prompt like this. gpt-4 answers this correctly without any tuning:
you have 2 boxes, box a and box b. Box a has 3 index cards, each with a giant letter printed in the middle
with a small caption below the letter. Based on the contents of Box A and Box B, infer what the blank
index card in Box B should most likely say
The contents of box a:
letter P, caption P:O
letter T, caption T:S
letter J, caption J:K
The contents of box b:
index card with O printed in the middle
index card with S printed in the middle
blank index card
- simonw 4y agoI interpreted "establish truthfulness of arbitrary statements" as referring to the challenge of telling things like if a statement like "mars has a population of 12 million" is truthful or not, given the available training data for the LLM.
- computerex 4y agoThat's also "solved". Yes models exhibit hallucinations, but chain of thought prompting and ReAct prompting improve accuracy on factual tasks greatly, particuarly in a data augmented generation setting. We are finding new and better ways to prompt language models. Scaling up the models even further will also improve performance.