3 ms·

I've spent the last decade in the most technical parts of this industry; it is my job to expunge this credulousness. It is indeed trivial to show that P(A|B) i

I've spent the last decade in the most technical parts of this industry; it is my job to expunge this credulousness. It is indeed trivial to show that P(A|B) is a poor model of "B => A" (and B causes A, and many other relata). Software engineers, philosophers and experimental scientists seem pretty good at seeing this --- people who "convert into" engineering are totally dumbfounded by it. P(A|B) becomes an increasingly 'useful' model of arbitrary relations as the implicit model P_m(A|B) grows to include all instances of A,B. That's what digitising all of human history and storing it in the weights of an LLM does. This all follows from basic stats you'd be taught in an applied statistics course; one never taken by most in the ML industry. (Note its still a broken model because there's an infinite number of novel instances of (A,B) pairs in most cases that cannot be modelled with this sort of inductive learning). Engineering at its heart, is a kind of pseudoscience (or, if you prefer: a magic trick). You find some heuristic which behaves as-if its the target under fragile but engineering-stable conditions. The problem with engineers who only have magic tricks in their toolkit is this credulousness. Homeopathy worked: you put people in beds and give them water and they recover (indeed, better than leaching them).

antonvs 3y ago
> people who "convert into" engineering are totally dumbfounded by it. How does that human failure to reason affect the evaluation of GPT's reasoning capabilities? You're being very dogmatic that anyone who has a different opinion than you are "desperate to not thinking clearly" or "gullible", but you're exhibiting errors in reasoning that are rather ironic given the context. Edit: You don't know how human reasoning works - no-one does. People tend to assume that our post-hoc conscious ability to "understand" the reasoning process must somehow relate to the actual operations the brain performs when reasoning. But that's not necessarily the case. Note that I'm not claiming that LLMs are equivalent to human in their reasoning ability: after all, in some cases they're functionally demonstrably superior, especially compared to the average human. But in other significant cases, they're certainly worse. The point is we shouldn't impose too many assumptions on how reasoning "should" work, and there seems to be a strong tendency to do that, which you're exhibiting.
- mjburgess 3y ago
  Maybe this is a clearer way of thinking about the problem: Imagine you call a friend for help on an essay (on any topic of your choice). But they have access to google and libgen and they're quick at looking. For awhile, on the call, you think your friend is a genius but then you get suspicious. How would you tell if they knew what they were talking about? NB. whether they do or not has nothing to do with whatever qs you come up with -- they do or they dont. And, indeed, people are easy to fool. You might ask, "why care if they know?" but then you have serious questions on things which matter, not just essays. And googling isnt good enough. Being able to weight information by expertise, engage in reasoning in which contradictions are impermissible, engage in causal inference, in abduction -- in imagining possibilities which have never occurred exactly before ------ these suddenly become vital. And your friend who is not a PhD in anything, let alone everything, suddenly becomes vastly more dangerous and insidious.
  - antonvs 3y ago
    > they do or they dont Only a Sith... Seriously, the idea that "reasoning" could be a binary property seems very disconnected from the reality of the situation. > reasoning in which contradictions are impermissible Every human on the planet would fail that test, including you. "Fooling apes" is much easier when the ape you need to fool is yourself.
  - jcparkyn 3y ago
    So from your analogy, you're saying the average high school student (approximately the "friend" in your scenario) is incapable of reasoning? Sure, they might be bad at it (and know nothing about formal methods), but most people's definition of reasoning is nowhere near that strict.
- freejazz 3y ago
  >How does that human failure to reason affect the evaluation of GPT's reasoning capabilities? Isn't that your argument? That not knowing how humans reason means that we can't say that GPT isn't reasoning?
visarga 3y ago
Maybe science has a formal process somewhere but it is mostly an open ended iterative exploration of ideas, especially around emerging fields.

antonvs 3y ago

> people who "convert into" engineering are totally dumbfounded by it. How does that human failure to reason affect the evaluation of GPT's reasoning capabilities? You're being very dogmatic that anyone who has a different opinion than you are "desperate to not thinking clearly" or "gullible", but you're exhibiting errors in reasoning that are rather ironic given the context. Edit: You don't know how human reasoning works - no-one does. People tend to assume that our post-hoc conscious ability to "understand" the reasoning process must somehow relate to the actual operations the brain performs when reasoning. But that's not necessarily the case. Note that I'm not claiming that LLMs are equivalent to human in their reasoning ability: after all, in some cases they're functionally demonstrably superior, especially compared to the average human. But in other significant cases, they're certainly worse. The point is we shouldn't impose too many assumptions on how reasoning "should" work, and there seems to be a strong tendency to do that, which you're exhibiting.

mjburgess 3y ago
Maybe this is a clearer way of thinking about the problem: Imagine you call a friend for help on an essay (on any topic of your choice). But they have access to google and libgen and they're quick at looking. For awhile, on the call, you think your friend is a genius but then you get suspicious. How would you tell if they knew what they were talking about? NB. whether they do or not has nothing to do with whatever qs you come up with -- they do or they dont. And, indeed, people are easy to fool. You might ask, "why care if they know?" but then you have serious questions on things which matter, not just essays. And googling isnt good enough. Being able to weight information by expertise, engage in reasoning in which contradictions are impermissible, engage in causal inference, in abduction -- in imagining possibilities which have never occurred exactly before ------ these suddenly become vital. And your friend who is not a PhD in anything, let alone everything, suddenly becomes vastly more dangerous and insidious.
- antonvs 3y ago
  > they do or they dont Only a Sith... Seriously, the idea that "reasoning" could be a binary property seems very disconnected from the reality of the situation. > reasoning in which contradictions are impermissible Every human on the planet would fail that test, including you. "Fooling apes" is much easier when the ape you need to fool is yourself.
- jcparkyn 3y ago
  So from your analogy, you're saying the average high school student (approximately the "friend" in your scenario) is incapable of reasoning? Sure, they might be bad at it (and know nothing about formal methods), but most people's definition of reasoning is nowhere near that strict.
freejazz 3y ago
>How does that human failure to reason affect the evaluation of GPT's reasoning capabilities? Isn't that your argument? That not knowing how humans reason means that we can't say that GPT isn't reasoning?

visarga 3y ago

Maybe science has a formal process somewhere but it is mostly an open ended iterative exploration of ideas, especially around emerging fields.