4 ms·
That's the problem with nondeterministic generative stuff: sometimes it get things right, and sometimes it doesn't and you cannot rely on any behavior.
by littlestymaar 3y ago
That's the problem with nondeterministic generative stuff: sometimes it get things right, and sometimes it doesn't and you cannot rely on any behavior.
- sigmoid10 3y agoI tried it 10 times and while the wording is different, the answer remained correct every time. I used the exact question from the comment above, nothing else. While determinism is a possible source of error, I find that in these cases people usually just use the wrong model on ChatGPT for whatever reason. And unless you set the temperature way too high, it is pretty unlikely that you will end up outside of correct responses as far as the internal world model is concerned. It just mixes up wording by using the next most likely tokens. So if the correct answer is "one", you might find "single" or "1" as similarly likely tokens, but not "two." For that to happen something must be seriously wrong either in the model or in the temperature setting.
- kenjackson 3y agoI got an answer with GPT-4 that is mostly wrong: "Sally has 2 sisters. Since each of her brothers has 2 sisters, that includes Sally and one additional sister." I think said, "wait, how many sisters does Sally have?" And then it answered it fully correctly.
- evanchisholm 3y agoKind of like humans?
- qayxc 3y agoHumans plural, yes. Humans as in single members of humankind, no. Ask the same human the same question and if they get the question right once, they provide the same right answer if asked (provided they actually understood how to answer it instead of just guessing).