5 ms·

1) doesn't make a whole lot of sense. A perfect understanding of something doesn't mean you can fix anything wrong with it. 3) has already happened. The AI cha

1) doesn't make a whole lot of sense. A perfect understanding of something doesn't mean you can fix anything wrong with it. 3) has already happened. The AI chat bots aren't very smart, but they're clearly capable of some degree of basic reasoning.

xvector 3y ago
I'd go as to far as saying GPT-4 is capable of fairly complex reasoning, at least in my experience tossing complicated questions and situations at ChatGPT Plus.
- andrewmutz 3y ago
  GPT4 is simultaneously smarter than and stupider than anyone you've ever met.
  - someplaceguy 3y ago
    I think that's a hilarious and accurate way of describing it :-)
- CSMastermind 3y ago
  When I was tutoring people in college for their computer science classes I was struck with how some people could do reasonably well on programming assignments and then when presented with anything even slightly novel would be completely unable to reason their way to a solution. A classic tell of this is people handling out of bounds errors in loops by trying to randomly add or subtract 1 from their for-loop parameters. I realized that they didn't have a mental model for what a loop did, they had simply memorized the syntax for a loop and were doing advanced pattern matching. Code repeats = write the for-loop syntax I've memorized. And then after seeing that fail with out of bounds exceptions, they learned a new rule: modify the loop parameters and see if that fixes the problem. When I think about how I write code, or I compare their approach to the other cohort of students I saw, it's a different process. I see in my mind's eye a type of 'machine' that performs the actions that I want to take place. I simulate running that machine in my mind and tweak its design until it works the way I want it to. Only then do I think about syntax and try to translate what's already happening in my mind into source code. I've seen people get shockingly far into software engineering careers using the pattern matching / guess and check approach. I've wondered if a lot of the handwringing you see on programming forums about the 'leetcode grind' is coming from people who do this pattern matching approach. To them it must seems like the only way to solve these problems is to simply train their internal pattern matching neural networks on huge amounts of examples. The code that I see GPT generate looks eerily similar to what I saw from those programmers. And that makes sense because I think that functionally they're doing the same thing. Only GPT does it at a superhuman level. That seems to me to indicate that there's something that at least some humans do with a mental model that our current LLMs lack. If someone figures out how to simulate those mental processes in a computer program I think we'll see a huge inflection point and that's what the original comment (as I read it) is referring to.
  - weard_beard 3y ago
    I predict it’s a visual/spatial form of what it already does with language. It won’t do it until it can see. There’s all kinds of other things it won’t do until it hears. And touches. Smell and taste might help too I guess!? As a byproduct it can also be taught truth is what it can verify with sensors.
  - Vecr 3y ago
    You mean that's now how everyone else programs? Obviously you do fuzz testing and static analysis and possibly some sort of theorem prover verification so you don't get too embarrassed.
  - ikt 3y ago
    that's me! i call myself a fake network engineer because there is simply too much info to retain, my brain just won't retain things i'm not constantly doing, so i have a complete understanding of maybe 20% of things but outside of that comparing to other configs and pattern matching are my main ways of solving problems and to be fair for my job (fixing network faults) like 90% of faults i've seen before and can fix, for 10% i can't i'm lucky that i have escalation points at the same time i do feel like pattern matching limits my growth, if i had a complete understanding of a majority of networking principals id be much higher up in my career
    - bdsa 3y ago
      If you compare yourself to colleagues at your level, particularly how _they_ would describe themselves, do you think they would be as self-aware? As long as it isn't making you feel like a complete fraud, this level of introspection is a good thing imo. "I know that I know nothing"
  - q7xvh97o2pDhNrh 3y ago
    > the handwringing you see on programming forums about the 'leetcode grind' In fairness and compassion to that crowd, a lot of it comes from the fact that a modern interview for a coveted FAANG job often requires 1-2 LC Medium (or Hard) problems cranked out in 45-60 minutes. Depending on the company and the org, the overall interview loop may well be multiple such one-hour sprints. It's quite a pressure-cooker of an interview setting. Given that, it's understandable why many people converge on memorizing and brute-force pattern-matching as their interview strategy — if they can just memorize enough, the odds are actually pretty decent. (And the payoff is not bad, either.)
- wredue 3y ago
  Chat-GPT is not capable of any reasoning, let alone “fairly complex”.
a_bonobo 3y ago
There's a wonderful paper on 1) from a few years ago: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... >Could a Neuroscientist Understand a Microprocessor? >We show that the [neuroscience experimental] approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor. This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data.
- mxkopy 3y ago
  This is a silly argument. It presupposes that meaningful knowledge about neurons is structured similarly to knowledge about microprocessors, which does not have to be the case since neurons aren’t structured like microprocessors.
- 0cf8612b2e1e 3y ago
  Nothing in biology is as regular and systematic as a microprocessor. Scientists study biology using round-about approaches because there is nothing else available.
  - zargon 3y ago
    This fact only makes the problem even worse.
- bschne 3y ago
  Which was inspired by this, "Could a Biologist Fix a Radio?" https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.pdf https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.p...
- mannykannot 3y ago
  This paper, and its precursor 'Could a Biologist Fix a Radio?' mentioned by bschne, come across as somewhat smug and wiser-than-thou. While they do a good job in showing the difficulty of gathering useful information about complex biological systems, they tend to underestimate what human ingenuity can do with just that information which can be gained experimentally. This was demonstrated recently by the rapid development of synthetic vaccines against Covid.
ChuckMcM 3y ago
The "event" I was trying to describe in #1 is that DNA and cellular mechanics are decoded to the point where one could do these things (repair, customization, Etc). I was not trying to specify all of the threshold requirements for reaching that point. Hopefully describing it that way makes more sense?
leptons 3y ago
Not sure that AI chat bots are actually reasoning in any basic way, so much as they are filtering out noise and presenting an output that might satisfy the input, but it's often quite wrong. It's still more of a parlor trick than it is intelligence or reason.
- icapybara 3y ago
  > Not sure that AI chat bots are actually reasoning in any basic way, so much as they are filtering out noise and presenting an output that might satisfy the input Not sure if there is a difference, for sufficiently abstract interpretations of “satisfy the input”
- LASR 3y ago
  There is no difference between simulated reasoning and reasoning.
  - leptons 3y ago
    Definition of Reasoning: "the action of thinking about something in a logical, sensible way." LLMs don't think, they extrapolate. They are a filter, not capable of thought or reason. You can't reason with an LLM but plenty of people have tried and it fooled them well enough.
    - xkcd1963 3y ago
      Totally! I'm aghasted of how many people think LLMs are capable of reasoning and thoughts. They would argue "but brains are just that, a big filter".LLMs don't get even close to what unicelulars can handle, and they don't have a brain.
irrational 3y ago
I interpret 3 as the chat bot requires no prompt from a human. Imagine the chat bot wakes you up in the middle of the night saying, "Hey, sorry to disturb you, but I was thinking... wouldn't eliminating all humans help to reverse the climate crisis more quickly? Just a thought." That is what I think of as number 3 (not actually wanting to kill off humans, but it is thinking on its own, without any prompts and then choosing to share what it has been thinking about with a human).
- LASR 3y ago
  Don’t we already have this? If you prompt GPT to “care” about a human and then keep sending it observations, it does a pretty good job.
  - irrational 3y ago
    But there is still an initial prompt. I think number 3 is all about no initial prompt. The AI reaches out, not because it has been programmed or prompted to do so, but because it wants to do so.

xvector 3y ago

I'd go as to far as saying GPT-4 is capable of fairly complex reasoning, at least in my experience tossing complicated questions and situations at ChatGPT Plus.

andrewmutz 3y ago
GPT4 is simultaneously smarter than and stupider than anyone you've ever met.
- someplaceguy 3y ago
  I think that's a hilarious and accurate way of describing it :-)
CSMastermind 3y ago
When I was tutoring people in college for their computer science classes I was struck with how some people could do reasonably well on programming assignments and then when presented with anything even slightly novel would be completely unable to reason their way to a solution. A classic tell of this is people handling out of bounds errors in loops by trying to randomly add or subtract 1 from their for-loop parameters. I realized that they didn't have a mental model for what a loop did, they had simply memorized the syntax for a loop and were doing advanced pattern matching. Code repeats = write the for-loop syntax I've memorized. And then after seeing that fail with out of bounds exceptions, they learned a new rule: modify the loop parameters and see if that fixes the problem. When I think about how I write code, or I compare their approach to the other cohort of students I saw, it's a different process. I see in my mind's eye a type of 'machine' that performs the actions that I want to take place. I simulate running that machine in my mind and tweak its design until it works the way I want it to. Only then do I think about syntax and try to translate what's already happening in my mind into source code. I've seen people get shockingly far into software engineering careers using the pattern matching / guess and check approach. I've wondered if a lot of the handwringing you see on programming forums about the 'leetcode grind' is coming from people who do this pattern matching approach. To them it must seems like the only way to solve these problems is to simply train their internal pattern matching neural networks on huge amounts of examples. The code that I see GPT generate looks eerily similar to what I saw from those programmers. And that makes sense because I think that functionally they're doing the same thing. Only GPT does it at a superhuman level. That seems to me to indicate that there's something that at least some humans do with a mental model that our current LLMs lack. If someone figures out how to simulate those mental processes in a computer program I think we'll see a huge inflection point and that's what the original comment (as I read it) is referring to.
- weard_beard 3y ago
  I predict it’s a visual/spatial form of what it already does with language. It won’t do it until it can see. There’s all kinds of other things it won’t do until it hears. And touches. Smell and taste might help too I guess!? As a byproduct it can also be taught truth is what it can verify with sensors.
- Vecr 3y ago
  You mean that's now how everyone else programs? Obviously you do fuzz testing and static analysis and possibly some sort of theorem prover verification so you don't get too embarrassed.
- ikt 3y ago
  that's me! i call myself a fake network engineer because there is simply too much info to retain, my brain just won't retain things i'm not constantly doing, so i have a complete understanding of maybe 20% of things but outside of that comparing to other configs and pattern matching are my main ways of solving problems and to be fair for my job (fixing network faults) like 90% of faults i've seen before and can fix, for 10% i can't i'm lucky that i have escalation points at the same time i do feel like pattern matching limits my growth, if i had a complete understanding of a majority of networking principals id be much higher up in my career
  - bdsa 3y ago
    If you compare yourself to colleagues at your level, particularly how _they_ would describe themselves, do you think they would be as self-aware? As long as it isn't making you feel like a complete fraud, this level of introspection is a good thing imo. "I know that I know nothing"
- q7xvh97o2pDhNrh 3y ago
  > the handwringing you see on programming forums about the 'leetcode grind' In fairness and compassion to that crowd, a lot of it comes from the fact that a modern interview for a coveted FAANG job often requires 1-2 LC Medium (or Hard) problems cranked out in 45-60 minutes. Depending on the company and the org, the overall interview loop may well be multiple such one-hour sprints. It's quite a pressure-cooker of an interview setting. Given that, it's understandable why many people converge on memorizing and brute-force pattern-matching as their interview strategy — if they can just memorize enough, the odds are actually pretty decent. (And the payoff is not bad, either.)
wredue 3y ago
Chat-GPT is not capable of any reasoning, let alone “fairly complex”.

a_bonobo 3y ago

There's a wonderful paper on 1) from a few years ago: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... >Could a Neuroscientist Understand a Microprocessor? >We show that the [neuroscience experimental] approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor. This suggests current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems, regardless of the amount of data.

mxkopy 3y ago
This is a silly argument. It presupposes that meaningful knowledge about neurons is structured similarly to knowledge about microprocessors, which does not have to be the case since neurons aren’t structured like microprocessors.
0cf8612b2e1e 3y ago
Nothing in biology is as regular and systematic as a microprocessor. Scientists study biology using round-about approaches because there is nothing else available.
- zargon 3y ago
  This fact only makes the problem even worse.
bschne 3y ago
Which was inspired by this, "Could a Biologist Fix a Radio?" https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.pdf https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.p...
mannykannot 3y ago
This paper, and its precursor 'Could a Biologist Fix a Radio?' mentioned by bschne, come across as somewhat smug and wiser-than-thou. While they do a good job in showing the difficulty of gathering useful information about complex biological systems, they tend to underestimate what human ingenuity can do with just that information which can be gained experimentally. This was demonstrated recently by the rapid development of synthetic vaccines against Covid.

ChuckMcM 3y ago

The "event" I was trying to describe in #1 is that DNA and cellular mechanics are decoded to the point where one could do these things (repair, customization, Etc). I was not trying to specify all of the threshold requirements for reaching that point. Hopefully describing it that way makes more sense?

leptons 3y ago

Not sure that AI chat bots are actually reasoning in any basic way, so much as they are filtering out noise and presenting an output that might satisfy the input, but it's often quite wrong. It's still more of a parlor trick than it is intelligence or reason.

icapybara 3y ago
> Not sure that AI chat bots are actually reasoning in any basic way, so much as they are filtering out noise and presenting an output that might satisfy the input Not sure if there is a difference, for sufficiently abstract interpretations of “satisfy the input”
LASR 3y ago
There is no difference between simulated reasoning and reasoning.
- leptons 3y ago
  Definition of Reasoning: "the action of thinking about something in a logical, sensible way." LLMs don't think, they extrapolate. They are a filter, not capable of thought or reason. You can't reason with an LLM but plenty of people have tried and it fooled them well enough.
  - xkcd1963 3y ago
    Totally! I'm aghasted of how many people think LLMs are capable of reasoning and thoughts. They would argue "but brains are just that, a big filter".LLMs don't get even close to what unicelulars can handle, and they don't have a brain.

irrational 3y ago

I interpret 3 as the chat bot requires no prompt from a human. Imagine the chat bot wakes you up in the middle of the night saying, "Hey, sorry to disturb you, but I was thinking... wouldn't eliminating all humans help to reverse the climate crisis more quickly? Just a thought." That is what I think of as number 3 (not actually wanting to kill off humans, but it is thinking on its own, without any prompts and then choosing to share what it has been thinking about with a human).

LASR 3y ago
Don’t we already have this? If you prompt GPT to “care” about a human and then keep sending it observations, it does a pretty good job.
- irrational 3y ago
  But there is still an initial prompt. I think number 3 is all about no initial prompt. The AI reaches out, not because it has been programmed or prompted to do so, but because it wants to do so.