4 ms·

Wow, the question posed to the neural nets (and their inability to respond) really gets to the heart of something that I've tried to articulate to others: that

by Kapura 3y ago

Wow, the question posed to the neural nets (and their inability to respond) really gets to the heart of something that I've tried to articulate to others: that ML cannot conceptualize of things in the abstract like people can. They cannot offer reasons, a train of thought like a person; they respond essentially "on instinct," and folks should be wary of the output of something like ChatGPT. Great article.

viraptor 3y ago
> They cannot offer reasons, a train of thought like a person That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.
huijzer 3y ago
I agree with you. It feels like a clever human on "fast thinking" mode, as Kahneman would call it. So when I ask a programming question, it feels like a master student answers the first thing that comes to mind. If you ask an explanation, the first explanation that comes to mind is blurted out.
- Smaug123 3y ago
  (Which is why prompt engineering is a thing! The art/science of phrasing the prompt so that the immediate blurted response is more likely to be correct.)
  - emmender1 3y ago
    prompt-engineering to get the answer is analogous to prompting the fortune teller to tell your future - ie, you have an intended answer that you expect (or wish for in the case of a fortune teller), and you tweak the prompts to get there.
    - Smaug123 3y ago
      Well, you might not know the answer, even if you know some properties the answer must have and some processes that might lead to the answer.
- wincy 3y ago
  I asked it about string formatting deduplication and it suggested using a HashSet since it deduplicates strings, and I’m like “are you sure that’s the best way to do this?” Then it apologized and gave a much more “standard” way the second time I asked. You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.
agucova 3y ago
Note that: > ML cannot conceptualize of things in the abstract like people can And: > They cannot offer reasons, a train of thought like a person Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/ https://thegradient.pub/othello/). What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent. The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.

by Kapura 3y ago

viraptor 3y ago

> They cannot offer reasons, a train of thought like a person That's not correct. Try asking a question that requires multiple steps of reasoning and add ", think step by step" to the prompt. This not only changes the output, but also often improves the quality of the result... like you'd expect it to happen with people.

huijzer 3y ago

I agree with you. It feels like a clever human on "fast thinking" mode, as Kahneman would call it. So when I ask a programming question, it feels like a master student answers the first thing that comes to mind. If you ask an explanation, the first explanation that comes to mind is blurted out.

Smaug123 3y ago
(Which is why prompt engineering is a thing! The art/science of phrasing the prompt so that the immediate blurted response is more likely to be correct.)
- emmender1 3y ago
  prompt-engineering to get the answer is analogous to prompting the fortune teller to tell your future - ie, you have an intended answer that you expect (or wish for in the case of a fortune teller), and you tweak the prompts to get there.
  - Smaug123 3y ago
    Well, you might not know the answer, even if you know some properties the answer must have and some processes that might lead to the answer.
wincy 3y ago
I asked it about string formatting deduplication and it suggested using a HashSet since it deduplicates strings, and I’m like “are you sure that’s the best way to do this?” Then it apologized and gave a much more “standard” way the second time I asked. You definitely need to know a little and be able to push back, it feels like. But it’s been an absolute champ in describing why things are going wrong in a general sense when I’ve been having issues, especially with generics and templates in C#.

agucova 3y ago

Note that: > ML cannot conceptualize of things in the abstract like people can And: > They cannot offer reasons, a train of thought like a person Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/ https://thegradient.pub/othello/). What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent. The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.