3 ms·

Maps well to Kahneman's "Thinking Fast and Slow" framework system 1 thinking for early layer processing of uncertainty in LLMs. quick, intuitive decisions, foc

Maps well to Kahneman's "Thinking Fast and Slow" framework system 1 thinking for early layer processing of uncertainty in LLMs. quick, intuitive decisions, focuses on uncertainty, happens in early transformer layers. system 2 thinking for later layer processing of empowerment (selecting elements that maximize future possibilities). strategic, deliberate evaluation, considering long-term possibilities, happens in later layers. system 1 = 4o/llama 3.1 system 1 + system 2 = o1/r1 reasoning models empowerment calculation seems possibly oversimplified - assumes a static value for elements over a dynamic context-dependent empowerment interesting that higher temperatures improved performance slightly for system 1 models although they still made decisions before empowerment information could influence them edit: removed the word "novel". The paper shows early-layer processing of uncertainty vs later-layer processing of empowerment.

optimalsolver 2y ago
> quick, intuitive decisions, focuses on what's novel/unknown Wouldn't it be the exact opposite? That is, novel stimuli requires more extensive processing at higher levels of cognition.
- deleted 2y ago
  [deleted]
- hinkley 2y ago
  Requires != receives. Knee-jerk reactions to novel circumstances get people into trouble all the time. Particularly in impulsive people, who find their actions sometimes don’t align with their values, to the point they can’t be friends with people who are particularly judgemental about actions versus words and won’t accept apologies. It is a lot of work to retrain your lizard brain to not react in unpleasant ways when the shit hits the fan or you’re exhausted, and the default answer to everything is “no”. There are ways I show up in a crisis that are exemplary, and others that I’m not proud of. Understanding why doesn’t fix it. It’s just step 1, like in AA.
hinkley 2y ago
Some people have been trying to claim this book is based on faulty research but I definitely know people who don’t believe they work the way the book describes but the only sane explanation for their behavior is that either they do or I know an inordinate number of pathological liars. And I know some neurodiverse people who absolutely believe this is why they are like this. And that’s always a problem with studies. Some things are 10% of the time, it works every time. If you test the general public for mental acuity and caffeine you’re going to see undiagnosed ADHD people benefitting, or find they’re already consuming alarming quantities of the stuff (self medication).
fudged71 2y ago
Stanovich proposes a three tier model http://keithstanovich.com/Site/Research_on_Reasoning_files/Stanovich_Two_MInds.pdf http://keithstanovich.com/Site/Research_on_Reasoning_files/S...
- nradov 2y ago
  Modeling an analog system like human cognition into any number of discrete tiers is inherently kind of arbitrary and unscientific. I doubt you could ever prove experimentally that all human thinking works through exactly two or three or ten or whatever number of tiers. But it's at least possible that a three-tier model facilitates building AI software which is "good enough" for many practical use cases.
  - User23 2y ago
    Funny you should say that because an American guy did that a hundred years ago and nailed it. He divided reasoning into the two categories corollarial and theorematic.
    - nradov 2y ago
      What is the experimental evidence to show that he nailed it?
      - User23 2y ago
        Do you want to try asking a question that isn't a category mistake?
    - westurner 2y ago
      Charles Sanders Peirce > Pragmatism > Theory of inquiry > Scientific Method: https://en.wikipedia.org/wiki/Charles_Sanders_Peirce#Scientific_method https://en.wikipedia.org/wiki/Charles_Sanders_Peirce#Scienti... Peirce’s Deductive Logic: https://plato.stanford.edu/entries/peirce-logic/ https://plato.stanford.edu/entries/peirce-logic/ Scientific Method > 2. Historical Review: Aristotle to Mill https://plato.stanford.edu/entries/scientific-method/#HisRevAriMil https://plato.stanford.edu/entries/scientific-method/#HisRev... Scientific Method: https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Scientific_method Reproducibility: https://en.wikipedia.org/wiki/Reproducibility https://en.wikipedia.org/wiki/Reproducibility Replication crisis: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis TFS > Replication crisis https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow "Language models can explain neurons in language models" https://news.ycombinator.com/item?id=35879007 https://news.ycombinator.com/item?id=35879007 : Lateralization of brain function: https://en.wikipedia.org/wiki/Lateralization_of_brain_function https://en.wikipedia.org/wiki/Lateralization_of_brain_functi...
svnt 2y ago
Predicting language tokens but without logic probably does not map well at all to system 1 thinking, except where fast=conversation.
- Jimmc414 2y ago
  the system 1/2 analogy is obviously imperfect for token prediction, but the paper does provide evidence that uncertainty processing in early layers (layer 2) affects decisions before empowerment processing in later layers (layer 72) can influence them. quick decisions based on uncertainty before deeper strategic evaluation does echo how fast thinking can override slow thinking, even if the mechanisms are substantially different.
nonameiguess 2y ago
I think it goes beyond that. When we say animals learn and humans in particular generate new knowledge by exploration, it's not just system I versus system II thinking. You might legitimately not be able to answer question just by thinking at all. You have to go out in the world and find the answer. It might be as simple and walking around a building to see what's on the other side or it might as complicated as spending a few thousand years learning to build particle accelerators and sensors that can infer what is happening at a subatomic level in these, but you can't just "generate" an answer with sheer brainpower. This is a lot older than Kahneman. It's the classical split between rationalists and empiricists. AI maximalists and x-risk death cult fanatics are effectively being rationalists. They don't believe a sufficiently intelligent entity will be bound by the need to do science. It can, by sheer force of thought, figure out the answer to any question, develop a perfect course of action to achieve any goal, as quickly as electricity can traverse its own internal circuits. It's interesting to see the constraints on how they can even study this in the paper, though. You can only ever ask a computing agent to do what it even can do. If you want it to explore, it needs a toy world that maybe digitally encodes a reasonable stripped-down representation of the real world, but without a body, sensors, and full autonomy, it can't ever truly explore the real real world.
- Qwertious 2y ago
  >it can't ever truly explore the real real world. Would you believe that computers are made out of atoms, and that computer programs modify those atoms? Via electrons, I believe.

optimalsolver 2y ago

> quick, intuitive decisions, focuses on what's novel/unknown Wouldn't it be the exact opposite? That is, novel stimuli requires more extensive processing at higher levels of cognition.

deleted 2y ago
[deleted]
hinkley 2y ago
Requires != receives. Knee-jerk reactions to novel circumstances get people into trouble all the time. Particularly in impulsive people, who find their actions sometimes don’t align with their values, to the point they can’t be friends with people who are particularly judgemental about actions versus words and won’t accept apologies. It is a lot of work to retrain your lizard brain to not react in unpleasant ways when the shit hits the fan or you’re exhausted, and the default answer to everything is “no”. There are ways I show up in a crisis that are exemplary, and others that I’m not proud of. Understanding why doesn’t fix it. It’s just step 1, like in AA.

hinkley 2y ago

Some people have been trying to claim this book is based on faulty research but I definitely know people who don’t believe they work the way the book describes but the only sane explanation for their behavior is that either they do or I know an inordinate number of pathological liars. And I know some neurodiverse people who absolutely believe this is why they are like this. And that’s always a problem with studies. Some things are 10% of the time, it works every time. If you test the general public for mental acuity and caffeine you’re going to see undiagnosed ADHD people benefitting, or find they’re already consuming alarming quantities of the stuff (self medication).

fudged71 2y ago

Stanovich proposes a three tier model http://keithstanovich.com/Site/Research_on_Reasoning_files/Stanovich_Two_MInds.pdf http://keithstanovich.com/Site/Research_on_Reasoning_files/S...

nradov 2y ago
Modeling an analog system like human cognition into any number of discrete tiers is inherently kind of arbitrary and unscientific. I doubt you could ever prove experimentally that all human thinking works through exactly two or three or ten or whatever number of tiers. But it's at least possible that a three-tier model facilitates building AI software which is "good enough" for many practical use cases.
- User23 2y ago
  Funny you should say that because an American guy did that a hundred years ago and nailed it. He divided reasoning into the two categories corollarial and theorematic.
  - nradov 2y ago
    What is the experimental evidence to show that he nailed it?
    - User23 2y ago
      Do you want to try asking a question that isn't a category mistake?
  - westurner 2y ago
    Charles Sanders Peirce > Pragmatism > Theory of inquiry > Scientific Method: https://en.wikipedia.org/wiki/Charles_Sanders_Peirce#Scientific_method https://en.wikipedia.org/wiki/Charles_Sanders_Peirce#Scienti... Peirce’s Deductive Logic: https://plato.stanford.edu/entries/peirce-logic/ https://plato.stanford.edu/entries/peirce-logic/ Scientific Method > 2. Historical Review: Aristotle to Mill https://plato.stanford.edu/entries/scientific-method/#HisRevAriMil https://plato.stanford.edu/entries/scientific-method/#HisRev... Scientific Method: https://en.wikipedia.org/wiki/Scientific_method https://en.wikipedia.org/wiki/Scientific_method Reproducibility: https://en.wikipedia.org/wiki/Reproducibility https://en.wikipedia.org/wiki/Reproducibility Replication crisis: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis TFS > Replication crisis https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow "Language models can explain neurons in language models" https://news.ycombinator.com/item?id=35879007 https://news.ycombinator.com/item?id=35879007 : Lateralization of brain function: https://en.wikipedia.org/wiki/Lateralization_of_brain_function https://en.wikipedia.org/wiki/Lateralization_of_brain_functi...

svnt 2y ago

Predicting language tokens but without logic probably does not map well at all to system 1 thinking, except where fast=conversation.

Jimmc414 2y ago
the system 1/2 analogy is obviously imperfect for token prediction, but the paper does provide evidence that uncertainty processing in early layers (layer 2) affects decisions before empowerment processing in later layers (layer 72) can influence them. quick decisions based on uncertainty before deeper strategic evaluation does echo how fast thinking can override slow thinking, even if the mechanisms are substantially different.

nonameiguess 2y ago

I think it goes beyond that. When we say animals learn and humans in particular generate new knowledge by exploration, it's not just system I versus system II thinking. You might legitimately not be able to answer question just by thinking at all. You have to go out in the world and find the answer. It might be as simple and walking around a building to see what's on the other side or it might as complicated as spending a few thousand years learning to build particle accelerators and sensors that can infer what is happening at a subatomic level in these, but you can't just "generate" an answer with sheer brainpower. This is a lot older than Kahneman. It's the classical split between rationalists and empiricists. AI maximalists and x-risk death cult fanatics are effectively being rationalists. They don't believe a sufficiently intelligent entity will be bound by the need to do science. It can, by sheer force of thought, figure out the answer to any question, develop a perfect course of action to achieve any goal, as quickly as electricity can traverse its own internal circuits. It's interesting to see the constraints on how they can even study this in the paper, though. You can only ever ask a computing agent to do what it even can do. If you want it to explore, it needs a toy world that maybe digitally encodes a reasonable stripped-down representation of the real world, but without a body, sensors, and full autonomy, it can't ever truly explore the real real world.

Qwertious 2y ago
>it can't ever truly explore the real real world. Would you believe that computers are made out of atoms, and that computer programs modify those atoms? Via electrons, I believe.