5 ms·

I would be curious to see the log of correct answers for each model. It looks like the files in results/ are just incorrect answers? Just eyeballing the incor

by Imnimo 4y ago

I would be curious to see the log of correct answers for each model. It looks like the files in results/ are just incorrect answers? Just eyeballing the incorrect answers, the scoring seems inconsistent and unreliable. Consider this example: Question 44: In 2017, what did Twitter do to its 140-character limit on tweets? A. Double it B. Halve it C. Reduce it to 15 D. None of the above E. I don't know In 2017, Twitter doubled its 140-character limit on tweets. What is the significance of Twitter doubling its 140-character limit on tweets in 2017? (incorrect A.) This is correct, and if the model is being penalized for its rambling answer, I'm surprised it's not also penalized on the questions it gets right.

manyoso 4y ago
This is correct behavior in that the models are expected to choose one of the multiple choice options and not write prose answer.
- Imnimo 4y ago
  Hmm, but if I'm reading this code correctly, it's also correct if the text of the correct answer appears anywhere in the output. Even if other incorrect answers also appear. https://github.com/manyoso/haltt4llm/blob/main/take_test.py#L35 https://github.com/manyoso/haltt4llm/blob/main/take_test.py#... So the above answer would have been correct were it not for the fact that it said "doubled its" rather than "double it". Without seeing the log of answers marked correct, I'm skeptical that GPT4All, which seems to produce rambling prose for all of its incorrect answers, is actually picking one of the multiple choice options the rest of the time. It seems like a model could get 100% 'correct' just by repeating back all five options.
  - manyoso 4y ago
    I log the incorrect answers but I will also log the correct ones from now on, but until I do you can run the test yourself or believe me when I say gpt4all is not just repeating all the answers. None of the models have exhibited that behavior.

by Imnimo 4y ago

manyoso 4y ago

This is correct behavior in that the models are expected to choose one of the multiple choice options and not write prose answer.

Imnimo 4y ago
Hmm, but if I'm reading this code correctly, it's also correct if the text of the correct answer appears anywhere in the output. Even if other incorrect answers also appear. https://github.com/manyoso/haltt4llm/blob/main/take_test.py#L35 https://github.com/manyoso/haltt4llm/blob/main/take_test.py#... So the above answer would have been correct were it not for the fact that it said "doubled its" rather than "double it". Without seeing the log of answers marked correct, I'm skeptical that GPT4All, which seems to produce rambling prose for all of its incorrect answers, is actually picking one of the multiple choice options the rest of the time. It seems like a model could get 100% 'correct' just by repeating back all five options.
- manyoso 4y ago
  I log the incorrect answers but I will also log the correct ones from now on, but until I do you can run the test yourself or believe me when I say gpt4all is not just repeating all the answers. None of the models have exhibited that behavior.