3 ms·
This is correct behavior in that the models are expected to choose one of the multiple choice options and not write prose answer.
by manyoso 4y ago
This is correct behavior in that the models are expected to choose one of the multiple choice options and not write prose answer.
- Imnimo 4y agoHmm, but if I'm reading this code correctly, it's also correct if the text of the correct answer appears anywhere in the output. Even if other incorrect answers also appear. https://github.com/manyoso/haltt4llm/blob/main/take_test.py#L35 https://github.com/manyoso/haltt4llm/blob/main/take_test.py#... So the above answer would have been correct were it not for the fact that it said "doubled its" rather than "double it". Without seeing the log of answers marked correct, I'm skeptical that GPT4All, which seems to produce rambling prose for all of its incorrect answers, is actually picking one of the multiple choice options the rest of the time. It seems like a model could get 100% 'correct' just by repeating back all five options.
- manyoso 4y agoI log the incorrect answers but I will also log the correct ones from now on, but until I do you can run the test yourself or believe me when I say gpt4all is not just repeating all the answers. None of the models have exhibited that behavior.