4 ms·
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt. With the simpler prompt, all the answer
by adrian_b 3y ago
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt.
With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
- Joeri 3y agoThere seems to be a maximum amount of reasoning llm’s can do per token (per unit of computation). If you prompt it to use more tokens before it outputs the final answer (think step by step, check your answer, …) it becomes smarter. People have lucked into different prompting strategies to get it to do this, but there probably are more. Ultimately I feel it is fairer to benchmark llm’s by what they can be prompted into. After all, we let people carefully work through a problem during exams so it seems fair to hold llm’s to the same standard.
- kaashif 3y agoIf we're under attack, launch the nukes. Oh wait, forgot something: Think it through step by step. Phew, close one.
- 0xDEF 3y agoI asked GPT-4 through both ChatGPT and the API and it responded correctly without any fiddling. I keep seeing comments and posts on HN that significantly downplay GPT-4's capabilities. Are people actually using GPT-4 or are they using a 3rd party service that claims to be GPT-4? I got: >Sally has 3 brothers, and each of those brothers has 2 sisters. One of those sisters is Sally herself, and the other one is Sally's sister. So, Sally has 1 sister.
- junofan 3y agoHere’s what ChatGPT GPT-4 gives me: > Sally has 2 sisters. Each of her 3 brothers has 2 sisters, and those sisters would be Sally and her 2 sisters.
- facu17y 3y agoEvery now and then GPT4 outputs a wrong answer. It's impossible to build a reliable product on top of GPT4 that is not a simple chat bot.