4 ms·
The quiz asks how likely GPT-4 is to solve a given question correctly, and the user has to enter their guess as a number between 0 and 1 (a probability). Howeve
by layer8 3y ago
The quiz asks how likely GPT-4 is to solve a given question correctly, and the user has to enter their guess as a number between 0 and 1 (a probability). However, the site then doesn’t provide the actual probability of GPT-4 providing a correct answer, to compare the guess against. This is, presumably, because there’s no practical way to determine that probablity with much precision. Also, in practice “correct” isn’t always black and white, e.g. there’s “not wrong but also not completely right”.
So I’m a bit confused about the whole premise.
- croes 3y agoIsn't your input your confidence that GPT-4 gives the correct answer, like I'm 100% sure GPT-4 gives the right answer for the capital of France but I'm only 50% sure it gets the ASCII art correct, and 0% for another question which means 100% GPT is wrong? Because how could he know the probability of getting the correct answer? He just tried and then it's a yes or no. And if it's right/wrong shouldn't GPT right/wrong 100% of the time for the same question and the same version?
- layer8 3y agoRegarding your last question, ChatGPT isn’t deterministic. It can certainly give a correct answer one time and an incorrect answer another time, for the same prompt.
- croes 3y agoFor the same prompt and same version? That would be deal breaker if the same prompt gives different results. Would also make the whole prompt engineering thing pretty useless.
- claytonjy 3y agoYes, same prompt and same version. GPT-4 is especially stochastic, even with temperature=0.
- croes 3y agoSo all these comments claiming that an article about the GPT's responses is wrong are useless. There are also such comments on this article. https://news.ycombinator.com/item?id=37361521 https://news.ycombinator.com/item?id=37361521
- rytill 3y agoI wonder why that is. Maybe they’re A/B testing assistant prompts, or there’s noise built into the architecture somewhere.
- nomel 3y agoThere was a post about it a while back. From what I recall, it’s an efficiency/cost reduction side effect, but it could be made deterministic, if they wanted to eat the cost.
- whimsicalism 3y agoI firmly disagree with your latter two statements, but yes it is non deterministic
- capableweb 3y ago> Regarding your last question, ChatGPT isn’t deterministic. It can certainly give a correct answer one time and an incorrect answer another time, for the same prompt. It's supposed to be deterministic if you set temperature to 0.0, but seems that doesn't work well in GPT4 compared to earlier versions...
- Yiin 3y agoIt is not deterministic since text-davinci-001. https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
- croes 3y agoGood to know, I thought it was deterministic, especially because of all the comments on other articles where people wrote that the author was crazy for getting a different answer to the same question.
- layer8 3y ago> Isn't your input your confidence that GPT-4 gives the correct answer You may be right that that’s the intent, however what’s the point (other than collecting data about user confidences)? If I enter 0.3 and GPT provides a correct answer, then that doesn’t mean that the 0.3 was somehow wrong.
- jefftk 3y agoIn that case 0.3 would be more wrong than 0.4 and less wrong than 0.2. The closer your predictions are to reality over a bunch of questions, the better you understand reality.
- layer8 3y agoYou can’t really say that for a single data point. The 0.3 may be completely correct. Now, if you try ten times, things might be different.
- croes 3y agoIsn't the whole point to show you how right your confidence is about GPT's capabilities? At least the results are about the quiz taker and his confidence.
- whimsicalism 3y agoIt’s a noisy measurement of how right you were.
- kqr 3y agoYou're right. But if you enter 0.3 on average over 28 questions and the actual number of true answers differ by a lot from 8 then you have learned your general sense of GPT-4s abilities is uncalibrated.
- DavidSJ 3y agoYou can still determine the log likelihood of your predictions across the whole set of questions, even if you only have a single sampled response for each question.
- kqr 3y agoBrier ("quadratic") store is another popular way to evaluate ensembles of forecasts.
- DavidSJ 3y agoYes, and it can be simpler to calculate it, but it tends to have worse theoretical properties. In particular, with log likelihood we can sum the scores for a sequence of conditional probability estimates and we get the same score as if it were a single estimate for a single compound random variable. Brier score not only lacks this property, but you don't even necessarily maintain the ordering of such scores.
- andrewmutz 3y agoThe tool is testing your ability to predict whether or not GPT-4 can get a task done correctly. You are supposed to provide your confidence level that it can do a task or not. If you answer all the questions, at the end the tool will tell you how well calibrated your beliefs about GPT-4s capabilities are, and how that calibration compares to other users of the tool.
- aeternum 3y agoEven though this webpage seems like it is actually sending to GPT-4 it seems that it is not as I tried a few and get very different answers. Many that the webpage claims are wrong GPT-4 actually gets right. Maybe it's some recent changes, but the flight time question for example, I tried many and was never able to get GPT4 to return any incorrect answer.
- thomasfromcdnjs 3y agoI got a "Wrong" answer, as I sit here and listen to the worst rendition of "Happy Birthday" ever. On a more serious note, I'd be interested in a similar test that use's the auto agent stuff, can a sufficiently setup LLM answer any question.