3 ms·

Typical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well i

Typical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well in zero shot despite this. They undoubtedly work better with a. Chain of thought b. Feedback Gpt-3 performance on MultiArith goes from 18% zero shot to 93% with chain of thought and feedback (or 79% with just chain of thought) All of this is to say that complex chains will produce better results. Probably they cherry picked but that's definitely not the only thing going on

nl 4y ago
Chain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.
- famouswaffles 4y ago
  It can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.
  - nl 4y ago
    > the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.
faizshah 4y ago
What do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?
- famouswaffles 4y ago
  Feedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus
angarg12 4y ago
I'm definitely getting better at asking questions, but the tool still seems to have some very rough edges. Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better. The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing. After 3 tries I gave up, fixed the code myself, and started a new chat with this code. Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.

nl 4y ago

Chain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.

famouswaffles 4y ago
It can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.
- nl 4y ago
  > the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.

faizshah 4y ago

What do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?

famouswaffles 4y ago
Feedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus

angarg12 4y ago

I'm definitely getting better at asking questions, but the tool still seems to have some very rough edges. Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better. The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing. After 3 tries I gave up, fixed the code myself, and started a new chat with this code. Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.