4 ms·

It has a model, but it is not a rational model. This difference is something that often throws engineers off track when thinking about generative AI. LLMs mode

It has a model, but it is not a rational model. This difference is something that often throws engineers off track when thinking about generative AI. LLMs models work more like intuitions. They are able to make statements about a problem in context, but they are generated from ideas that "instinctively" make sense given the prior statements and learned corpus (similar to Daniel Kahneman's fast mode of thinking), not logically constructed. These models do not have the capability to build formal inferences of careful steps that try to validate those ideas avoiding contradictions. Those capabilities can be added outside the model to try to verify the generated text, but so far are not integrated in the learning process, and I don't think anyone knows how to do it.

SanderNL 3y ago
Rationality’s building blocks are themselves not rational. I don’t know where this idea came from that logical thought somehow springs into life fully formed at once. Logos? I find it more helpful to think of human thought as consisting of multitudes of little patterns, all wired up together to correlate but individually unrecognizable and certainly not traceable to some concrete part of a problem. At some unknown and slightly fuzzy point our dreamlike mentations start resembling some form of rational thought. But it’s a mirage that will fade again in time. Like how clouds suddenly and instantly look like a rabbit, it’s a trick. Thousands of patterns that are not rabbitlike in any way had to help that activation along the way. I think the trick is building so much margin, so much room between dreamlike mentations and “rationality” that the entity stays coherent most of the time under normal circumstances. I think this is vaguely what happens with moving from gpt3 to gpt4, it got some breathing room. Remember it is quite easy to trick a human into decoherence as well.
- TuringTest 3y ago
  > Rationality’s building blocks are themselves not rational. I don’t know where this idea came from that logical thought somehow springs into life fully formed at once. Certainly not from me :-P I'm fully aware that human rationality is one technique trained on top of our common diffuse thinking. Heck, we invented machines to perform rational steps for us without errors. Once you build a consistent rational system though, you can trust that it will always produce internally coherent knowledge (as long as no bugs external to the system are introduced). That behaviour requires algorithms, not statistical inference.
  - SanderNL 3y ago
    > Certainly not from me Oh for sure. I’m just aimlessly rambling at this point. > That behaviour requires algorithms, not statistical inference. I’m not sure that’s possible. “Rational” systems may be an oxymoron and/or only applicable in extremely narrow domains. Like calculators. I’m getting the vibe anything approaching generality will be by nature vague. I just hope we can increase the “illusion of rationality”-bandwidth so it coincides with our needs in the workforce.
    - TuringTest 3y ago
      Rational system is anything that follows a mathematically formal process. This server where we post our messages is a rational system, as well as any deterministic software. It's not a high bar to achieve. There may be limits to rationality (both for being incomplete to describe the world and for generating results that can't be demonstrated within themselves), but being rational by itself is nothing more than the capability to apply formal logic, i.e the capability to generate a sequence of sentences derived from the previous steps and a limited number of rules of inference.
      - SanderNL 3y ago
        Mathematics, rationality, logic, "formal process". They are like talking about "threads", "file descriptors" and "inodes". Fancy abstractions, but what are they abstracting away precisely? What are the roots they are trying to hide? Is the foundation of logic itself logical? Of course this degenerates into word-play quickly, but the problem itself isn't solved (or diminished) by word-play. > It's not a high bar to achieve. I'm not sure I agree. It's not trivial at all to produce a system that can support what you call logic and formal processes, both cognitively - you have to think it up in the first place and Turing and von Neumann and others were no slouches - and physically as the history of and continued development of computational circuitry shows. I think what you call rational or well-behaved systems are just mirages or shadows that pop into life once a specific pattern of fundamentally non-rational behaviours intersect just right. I, of course, am unhindered by any actual knowledge or competency in this domain so I wouldn't read too much into what my unhinged subconscious is spewing forth.
deleted 3y ago
[deleted]
moffkalast 3y ago
Yeah agreed, all thinking is "fast" as it were, although with self prompting plus tool use one gets closer to slow thinking that results in more rational reasoning (e.g. get the data from wolfram alpha, plug it into a calculator api, return results). No guarantee that it'll be rational though, much like in humans. If it had absolutely no model of the world it would be unable to dynamically reason about it at all which it very much does [citation needed I guess], so there's definitely something there.

SanderNL 3y ago

Rationality’s building blocks are themselves not rational. I don’t know where this idea came from that logical thought somehow springs into life fully formed at once. Logos? I find it more helpful to think of human thought as consisting of multitudes of little patterns, all wired up together to correlate but individually unrecognizable and certainly not traceable to some concrete part of a problem. At some unknown and slightly fuzzy point our dreamlike mentations start resembling some form of rational thought. But it’s a mirage that will fade again in time. Like how clouds suddenly and instantly look like a rabbit, it’s a trick. Thousands of patterns that are not rabbitlike in any way had to help that activation along the way. I think the trick is building so much margin, so much room between dreamlike mentations and “rationality” that the entity stays coherent most of the time under normal circumstances. I think this is vaguely what happens with moving from gpt3 to gpt4, it got some breathing room. Remember it is quite easy to trick a human into decoherence as well.

TuringTest 3y ago
> Rationality’s building blocks are themselves not rational. I don’t know where this idea came from that logical thought somehow springs into life fully formed at once. Certainly not from me :-P I'm fully aware that human rationality is one technique trained on top of our common diffuse thinking. Heck, we invented machines to perform rational steps for us without errors. Once you build a consistent rational system though, you can trust that it will always produce internally coherent knowledge (as long as no bugs external to the system are introduced). That behaviour requires algorithms, not statistical inference.
- SanderNL 3y ago
  > Certainly not from me Oh for sure. I’m just aimlessly rambling at this point. > That behaviour requires algorithms, not statistical inference. I’m not sure that’s possible. “Rational” systems may be an oxymoron and/or only applicable in extremely narrow domains. Like calculators. I’m getting the vibe anything approaching generality will be by nature vague. I just hope we can increase the “illusion of rationality”-bandwidth so it coincides with our needs in the workforce.
  - TuringTest 3y ago
    Rational system is anything that follows a mathematically formal process. This server where we post our messages is a rational system, as well as any deterministic software. It's not a high bar to achieve. There may be limits to rationality (both for being incomplete to describe the world and for generating results that can't be demonstrated within themselves), but being rational by itself is nothing more than the capability to apply formal logic, i.e the capability to generate a sequence of sentences derived from the previous steps and a limited number of rules of inference.
    - SanderNL 3y ago
      Mathematics, rationality, logic, "formal process". They are like talking about "threads", "file descriptors" and "inodes". Fancy abstractions, but what are they abstracting away precisely? What are the roots they are trying to hide? Is the foundation of logic itself logical? Of course this degenerates into word-play quickly, but the problem itself isn't solved (or diminished) by word-play. > It's not a high bar to achieve. I'm not sure I agree. It's not trivial at all to produce a system that can support what you call logic and formal processes, both cognitively - you have to think it up in the first place and Turing and von Neumann and others were no slouches - and physically as the history of and continued development of computational circuitry shows. I think what you call rational or well-behaved systems are just mirages or shadows that pop into life once a specific pattern of fundamentally non-rational behaviours intersect just right. I, of course, am unhindered by any actual knowledge or competency in this domain so I wouldn't read too much into what my unhinged subconscious is spewing forth.

deleted 3y ago

[deleted]

moffkalast 3y ago

Yeah agreed, all thinking is "fast" as it were, although with self prompting plus tool use one gets closer to slow thinking that results in more rational reasoning (e.g. get the data from wolfram alpha, plug it into a calculator api, return results). No guarantee that it'll be rational though, much like in humans. If it had absolutely no model of the world it would be unable to dynamically reason about it at all which it very much does [citation needed I guess], so there's definitely something there.