3 ms·
When do you need chain-of-thought prompting for ChatGPT?
- belter 4y ago"...Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think step-by-step'' to each input query of MultiArith dataset, GPT-3's accuracy can be improved from 17.7\% to 78.7\%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and can generate CoT even without being instructed to do so... ...Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs..."
- amrb 4y agoI found to very difficult to set goals with gpt-turbo, maybe gpt4 will be a good test when I get access. Also I'm thinking would be interesting to train a LoRa, using synthetic prompts.
- inawarminister 4y agoHave you tried Langchain? The Search agent using GPT-turbo is really good in my experience. Now to replicate it using local LLAMA...
- nerdponx 4y agoIt's really weird to me that we now have research papers dedicated to exploring the emergent behavior of systems that other people built, but where the researchers can only speculate about what the builders actually did to build the thing. "Open" AI indeed.
- rmbyrro 4y agoI think the "Open" in OpenAI is about open access, not open source. I'm really glad they are "democratic" in releasing their products. I'd hate if I had to go through a "Talk to our salespeople" form ik order to use them.
- monsieurbanana 4y agoWith that definition every single big software company is open. Microsoft the most open of all, they don't care if you pirate Windows as it brings them more market share.
- throwawayapples 4y agoThen they wouldn't be nearly as good. You're training GPT for them for free (or even paying them for the privilege.)
- colordrops 4y agoNo, that's not what they meant. Read the history of its founding. They changed their charter after it was founded.
- jacquesm 4y agoIt might also be post/pre processing.
- jychang 4y agoSounds like talking to a kid (or any human being, really) about a complicated math homework problem. Making a kid say their reasoning step by step would increase accuracy.
- wiradikusuma 4y agoDoes anyone know a good tutorial for producing good prompts? I forgot the document name, but last time there was a "bible" for producing good results from text-to-image.
- splatzone 4y agoAre you referring to the DALLE-2 prompt book? https://dallery.gallery/wp-content/uploads/2022/07/The-DALL%C2%B7E-2-prompt-book.pdf https://dallery.gallery/wp-content/uploads/2022/07/The-DALL%...
- v9v 4y agoThis one was posted to HN a few weeks ago: https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/ https://lilianweng.github.io/posts/2023-03-15-prompt-enginee...
- chrispogeek 4y agoI think COT would still be useful for guiding the LLM to customized context. Most of these examples refer to vanilla arithmetic/logic reasoning COT.
- siavosh 4y agoIs COT considered a deep area of engineering/research or is it considered as simply a temporary hack that will be obsolete in the near or medium term?
- Ozzie_osman 4y agoDespite the fancy name (Chain Of Thought) and multiple arxiv papers, I'd definitely consider it a temporary hack that the models will overcome as they get more powerful and as they are better-trained to avoid hallucination. From this paper, it looks like ChatGPT may have already overcome it for some types of questions. What may be here to stay might be more task-specific prompting to break a problem down. For example, a variation might be "Let's think step by step, making sure we do or consider X before we do Y". By way of analogy, when my toddler was really young, if he couldn't figure something out, I'd ask him to break it down into steps (very similar to general Cot prompting). Now, he rarely needs that, but still needs more occasional, task-specific "prompting", eg "Let's think about the weather, and what we're going to be doing, before deciding what to wear".
- JieJie 4y agoIsn't breaking things down into steps always going to be a part of solving any problem? I think of how I was told to approach problems while developing software in this manner. Maybe the LLM gets so smart that it doesn't need to do this, but that I would definitely want to call a super-intelligence, because I sure can't solve big problems without breaking them down into smaller parts. But maybe you are on to something when you say there are some things that we've just done so much that we don't really have to think them through to accomplish them anymore. I don't know what that would say about an AI that was smart enough it could just spit out the right answer to literally any question we could devise. That would be quite a feat of engineering.
- vineyardmike 4y ago> But maybe you are on to something when you say there are some things that we've just done so much that we don't really have to think them through to accomplish them anymore. It’s like multiplication tables for thought. Using the “what to wear” example, you’ve essentially already just memorized what to wear for every occasion so you rarely need to think about it. Eg you know what is work appropriate, and if it’s raining or snowing or hot you’re still fine. But a wedding for a foreign culture you’re not familiar with in a city with weather you’re not familiar with, and you might not immediately know what to wear, so you have to break it down.
- boringuser2 4y agoAny type of additional explanation that isn't strictly necessary is currently painful on the GPT-4 plan.
- cubefox 4y agoBing Chat (Sydney) has 20 turns per conversation and 200 turns per day, free.
- raylad 4y agoWhich is the same as the paid ChatGPT plan when using GPT-4: 25 turns every 3 hours. Bing's free one is better in that sense in that you can use all 200 over a shorter timespan.
- Veen 4y agoIf you have access to GPT-4, you can use it in the OpenAI API Playground. It’s a bit more reliable than ChatGPT and I’ve not run into any completion limitations, although my use is strictly small-scale.
- snowe2010 4y agoMan how much talking are y'all doing with this thing? I haven't once even gotten close to hitting my limit.
- leobg 4y agoI think the ideas is that the LLM cannot think internally. It’s output is its thinking process. Especially with an auto regressive architecture like GPT, where each output token becomes part of the input. I imagine it like handing the LLM a piece of scratch paper.