3 ms·

I keep reading examples like this, but every time I used ChatGPT at work it ends up saving me a bit of time completing my task (at best), definitely nothing as

I keep reading examples like this, but every time I used ChatGPT at work it ends up saving me a bit of time completing my task (at best), definitely nothing as impressive. I wonder if these examples are heavily cherry-picked, or if I'm terrible at using the tool.

csomar 4y ago
It's a bit of both. The savings that ChatGPT did is impressive. And these examples are quite cherry-picked.
RcouF1uZ4gsC 4y ago
I suspect that there is a huge selection bias with regards to ChatGPT since reporting impressive results gets the most clicks.
- noahtallen 4y ago
  True, but I think this is totally fine and expected. We’re in an experimentation phase — a new discovery has been made, and everyone is experimenting, poking around, and testing it out. This is the only way we figure out what it will be good at (and what it will bad at!) Someone has an idea, tries to come up with a way for GPT to execute it and succeeds or fails. Along the way, we start developing better techniques and answers to the question, “what is it really useful for?” And in a while, we’ll know a lot more. But right now, we’re learning and discovering. IMO, this is a useful lens for viewing the news around Chat GPT. (Well, more useful than saying “it’s the future!” or “it’s useless” or “it’ll be the end of us!”)
famouswaffles 4y ago
Typical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well in zero shot despite this. They undoubtedly work better with a. Chain of thought b. Feedback Gpt-3 performance on MultiArith goes from 18% zero shot to 93% with chain of thought and feedback (or 79% with just chain of thought) All of this is to say that complex chains will produce better results. Probably they cherry picked but that's definitely not the only thing going on
- nl 4y ago
  Chain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.
  - famouswaffles 4y ago
    It can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.
    - nl 4y ago
      > the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.
- faizshah 4y ago
  What do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?
  - famouswaffles 4y ago
    Feedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus
- angarg12 4y ago
  I'm definitely getting better at asking questions, but the tool still seems to have some very rough edges. Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better. The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing. After 3 tries I gave up, fixed the code myself, and started a new chat with this code. Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.
nestorD 4y ago
Part of it that this is ChatGPT + tools (given via langchain). This opens the door to a lot of power. This is a good example: https://langchain.readthedocs.io/en/latest/modules/agents/getting_started.html https://langchain.readthedocs.io/en/latest/modules/agents/ge...
- BoorishBears 4y ago
  I spent a ton of time hacking together something like that only to just find langchain supports this: they definitely need to bring the tools use case to the top of the documentation since it's by far the most tedious part of what it offers to figure out on your own
mlejva 4y ago
It started as cherry picked but as you come up with better solutions and handling unexpected situations it becomes magically good at building things
noisycarlos 4y ago
Same here. It sounds really impressive until i try try it. Ive been trying it as help for coding a web app this week. Sometimes it's really helpful. But often it wouldn't work, I kept correcting it and telling it which errors i got. So it keeps giving me new suggestions that also didn't work, and it keeps adding code that doesn't help. So i end up with a bunch of long code from all the attempts that still doesn't work. So i have to fix it by hand anyway. Like i said, sometimes it's really good, but either I'm also bad at asking, or they have cherry-picked examples.
jhoelzel 4y ago
100% cherry-picked usually i cant even get the complete boilerplate of a project, because it losses its context. let alone the fact that you cant get around the token limit.

csomar 4y ago

It's a bit of both. The savings that ChatGPT did is impressive. And these examples are quite cherry-picked.

RcouF1uZ4gsC 4y ago

I suspect that there is a huge selection bias with regards to ChatGPT since reporting impressive results gets the most clicks.

noahtallen 4y ago
True, but I think this is totally fine and expected. We’re in an experimentation phase — a new discovery has been made, and everyone is experimenting, poking around, and testing it out. This is the only way we figure out what it will be good at (and what it will bad at!) Someone has an idea, tries to come up with a way for GPT to execute it and succeeds or fails. Along the way, we start developing better techniques and answers to the question, “what is it really useful for?” And in a while, we’ll know a lot more. But right now, we’re learning and discovering. IMO, this is a useful lens for viewing the news around Chat GPT. (Well, more useful than saying “it’s the future!” or “it’s useless” or “it’ll be the end of us!”)

famouswaffles 4y ago

Typical Zero shot Q/A is like coming up to a person, shouting a question and then taking the first then that comes to mind as the answer. LLMs work quite well in zero shot despite this. They undoubtedly work better with a. Chain of thought b. Feedback Gpt-3 performance on MultiArith goes from 18% zero shot to 93% with chain of thought and feedback (or 79% with just chain of thought) All of this is to say that complex chains will produce better results. Probably they cherry picked but that's definitely not the only thing going on

nl 4y ago
Chain-of-thought is still zero-shot performance. It's just prompting "thinking step by step" at the start of your prompt.
- famouswaffles 4y ago
  It can be that yes but the first CoT paper introduced few shot examples to induce the actual CoT and those generally produce better results. But yes you're right.
  - nl 4y ago
    > the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.
faizshah 4y ago
What do you mean by Feedback here? Is there a paper you’re referencing for Chain of Thought with feedback?
- famouswaffles 4y ago
  Feedback is a. pasting output of code back in b. For the benchmark inporovement I was talking about, they basically had gpt generate an algebraic and python solution of. Proceed if the output of the two matched, try again if it didn't. Guess I should have added another one - Consensus
angarg12 4y ago
I'm definitely getting better at asking questions, but the tool still seems to have some very rough edges. Here is my most egregious example. I asked ChatGPT to produce code for a very specific approach. It produced ok code, but I didn't like the approach, so I asked it to do it a different way. It produced a second function, but I ended up liking the first one better. The first code had a bug though, so I asked ChatGPT to fix it. It apologized and applied the fix... to the second code (which didn't make sense). I pointed this out, and it profusely apologized... and spitted out the same code again. I corrected it once more time, and did exactly the same thing. After 3 tries I gave up, fixed the code myself, and started a new chat with this code. Had it been a person I'd think it was a troll, a blithering idiot, or gaslighting me.

nestorD 4y ago

Part of it that this is ChatGPT + tools (given via langchain). This opens the door to a lot of power. This is a good example: https://langchain.readthedocs.io/en/latest/modules/agents/getting_started.html https://langchain.readthedocs.io/en/latest/modules/agents/ge...

BoorishBears 4y ago
I spent a ton of time hacking together something like that only to just find langchain supports this: they definitely need to bring the tools use case to the top of the documentation since it's by far the most tedious part of what it offers to figure out on your own

mlejva 4y ago

It started as cherry picked but as you come up with better solutions and handling unexpected situations it becomes magically good at building things

noisycarlos 4y ago

Same here. It sounds really impressive until i try try it. Ive been trying it as help for coding a web app this week. Sometimes it's really helpful. But often it wouldn't work, I kept correcting it and telling it which errors i got. So it keeps giving me new suggestions that also didn't work, and it keeps adding code that doesn't help. So i end up with a bunch of long code from all the attempts that still doesn't work. So i have to fix it by hand anyway. Like i said, sometimes it's really good, but either I'm also bad at asking, or they have cherry-picked examples.

jhoelzel 4y ago

100% cherry-picked usually i cant even get the complete boilerplate of a project, because it losses its context. let alone the fact that you cant get around the token limit.