4 ms·
Here's a trivial trick that considerably improves Gemini's ability to reason: ask it to explain the rationale behind its answer. To get to this, I asked him to
by ahmedfromtunis 3y ago
Here's a trivial trick that considerably improves Gemini's ability to reason: ask it to explain the rationale behind its answer.
To get to this, I asked him to predict the return value of some simple python function (for """def func(a,b): return a*3 + 2*b)""" what's func(2)).
without requiring the explanation, the answers were not only wrong but also changed between attempts for the same input.
Asking the model to provide an explanation, however, resulted in it giving the correct answer each time.
This method still allows for improvement in the quality of answers in more complex problems, although at a bit lower correctness rate.
- baq 3y agoThe original ‘think step by step’ also made this observation. Other models provide better answers if you tell them it’s important for your career or that you’ll tip them $200 if they’re right, or if you tell them that it’s allowed to not know… the preamble to get max performance gets long.
- dkuebric 3y agoI suspect all these strategies work on humans as well. Makes you think!
- zamadatix 3y agoIt's the other way around, that they work on humans is suspect to it working on LLMs. These LLMs are trained on predicting what the human training data would say next. Since these kind of things often result in the following responses in the data being different it results in the model learning that difference in what comes next as well.
- ttul 3y agoI was surprised to see in the paper that they appear to ask for an answer to a multiple-choice question, and only after getting the answer to the multiple-choice question do they ask the model for an explanation. The model will do much better if you ask it to explain its reasoning before it produces its answer to the multiple-choice question. The reason this works is it puts the reasoning into the context window. Transformers pay attention to everything in the context window and having a rationale sitting there - even if the model generates the rationale - helps the model do a better job of issuing the correct multiple choice answer.
- ahmedfromtunis 3y agoVery interesting. It's like asking the model to have an inner voice in its "mind" and to listen to what that inner voice is telling it.
- baq 3y agoOr like giving the machine time and tape to perform computation. Never tried that but an interesting experiment would be making it output irrelevant things - it could also help to get the right answer just due to the time and space.
- scotty79 3y agoFunnilly enough this also works with human children. When you ask them something their first attempt us a guess. Only when you ask them to answer and explain the answer they start thinking.