6 ms·

Evidence for entropy maximisation in human free choice behaviour

103 points by Gaessaki 6y ago

bArray 6y ago
There is some work that explores this from an AI perspective with relative success [1]. It turns out that you can create quite intelligent agents if they look to maximize future entropy, with interesting results [2]. It's still quite computationally expensive, but your normal search reduction tricks apply and you can get something computationally feasible. [1] https://arxiv.org/abs/1310.1863 https://arxiv.org/abs/1310.1863 [2] https://www.mdpi.com/1099-4300/16/5/2789 https://www.mdpi.com/1099-4300/16/5/2789
- sriku 6y ago
  An aside: this is great concept naming - "empowerment" for the channel capacity between sensors and actuation for an agent. It is at once relatable, fairly precise, memorable/recallable and therefore searchable .. instead of "PersonName's capacity" or something like that.
- medlyyy 6y ago
  That's a really great, interesting paper ([2]). Thanks for linking it here. This is a more recent (Dec 2020) paper by one of the authors on combining empowerment and extrinsic goals: https://ieeexplore.ieee.org/abstract/document/9284556 https://ieeexplore.ieee.org/abstract/document/9284556
PeterisP 6y ago
Isn't it pretty much the optimal behavior as evidenced e.g. by multi-armed bandit algorithms and explore-exploit balance in reinforcement learning?
- johbjo 6y ago
  Algorithms would try to maximize expected value. As far as I can tell, here it seems humans value choices over expected value. In other words, humans pay for the perception of freedom.
  - PeterisP 6y ago
    No, my point is that our research on algorithms dealing with uncertain rewards show that in such scenarios intentionally exploring choices (entropy maximization) is the optimal way to maximize long-term expected value. I.e. it's not that humans value choices over expected value, since valuing choices actually is the correct way to get larger expected value (with caveats such as how explore vs exploit tradeoff needs to be changed over time) - the message isn't that humans "pay for the perception of freedom" but that human evolved values, even seemingly irrational such as "need for perception of freedom" are actually close to mathematically optimal behavior.
    - aaronblohowiak 6y ago
      This seems like another way of stating "paying for optionality under uncertainty is generally worth it", which seems like a well-established result.
      - PeterisP 6y ago
        Well, testing whether the intuitive behavior of humans matches that or not is always an interesting result.
FlyingSaucer 6y ago
This topic is extremely interesting, and good to see that experiments support it. In AI lens: In a way, you can compare this to novelty seeking and intelligent exploration which is quite an active field in Artificial Life and game AI[1]. If you find this interesting: Jeff Clune, Kenneth Stanley and Joel Lehman conducted interesting related research. Also, isn't this somehow related to the Free Energy principle by Karl Friston? If you look at entropy maximization as a way to minimize surprises. [1] : https://arxiv.org/abs/1901.10995 https://arxiv.org/abs/1901.10995
- netizen-9748 6y ago
  Friston's principle of free energy was actually the first thing that came to mind on reading the title, it's an absolutely fascinating idea.
hkt 6y ago
On an intuitive level, this makes perfect sense. Assuming that entropy is, roughly speaking, novelty, that makes the calculation one about exploring new options for utility gains. I recall seeing a study (although not where) suggesting novelty seeking was a key hallmark of intelligence. Maybe this means the entropy-utility calibration drives their intelligence? (Alongside their actual material circumstances)
- alfonsodev 6y ago
  It's like the reverse of a boat breaking the ice while navigating the North Pole, I imagine you want to be moving towards entropy and leaving behind the opposite, in the edge between certainty and novelty. But also in loops, because we have to unlearn time to time, and break the ice, ok now I'm lost in my own metaphor :D
  - hkt 6y ago
    Lost in your metaphor as you might be, I like the idea of humans as moving towards entropy and leaving behind order..!
_nhynes 6y ago
Of course. That's why the first two versions of the Matrix failed.
johndoe42377 6y ago
There is no such thing as evidence for abstract concepts.
- macawfish 6y ago
  Prove it
  - johndoe42377 6y ago
    Easily. It is a type error. An attempt to apply inapplicable (orthogonal) concepts to one another. An evidence implies an observable phenomenon (whose description is a statement of fact). Merely a chain of reasoning is never an evidence, because it usually based on flawed premises or domain-specific logic. Now pay me.
- godelzilla 6y ago
  >These results show that free choice behaviour is well explained by an inference-based framework in which both utility and entropy are optimised. Yeah, I LOL'd at this conclusion in the abstract. This is a great example of economics not being science.
nemoniac 6y ago
Interesting paper. The exploration Vs exploitation payoff measured.
dmichulke 6y ago
aka Mobility heuristics A very strong heuristics that works well in many games (exceptions are usually very interesting games) and is the root to other heuristic concepts such as piece value, central positioning, "protected king", ... in Chess and similar concepts in, e.g., Starcraft. Also very easy to implement, for discrete turn-based games it's just the number of moves in a given state. http://ggp.stanford.edu/lectures/heuristics.pdf http://ggp.stanford.edu/lectures/heuristics.pdf
juskrey 6y ago
Look for Kelly criterion for the whole field on how to cook entropy
gglon 6y ago
Also see Casual Entropic Forces [1] by A. D. Wissner-Gross [2] and C. E. Freer [1] https://www.alexwg.org/publications/PhysRevLett_110-168702.pdf https://www.alexwg.org/publications/PhysRevLett_110-168702.p... [2] https://www.alexwg.org/ https://www.alexwg.org/
r-zip 6y ago
Their definition of entropy is missing a negative sign...I understand that this is a preprint but come on.
yottalove 6y ago
When my son has asked me what he should do about courses or employment or even vacation choices, I answer that he ought to choose the path that gives him the most choices. He has thanked me many time for that advice, which has resulted in a high-value path for him.
unabst 6y ago
Are we sure the availability of options equals entropy? It doesn't appear as though we all act to simply increase our options. Preferring options over reward may also constitute delayed gratification and sacrifice, which is another interesting can of worms, but can it be predicated in terms of just preference of options over reward when those are your only two artificial options? Human behavior appears to point towards the maximization of current order as an investment in power/potential to drive future entropy, as opposed to simply maximizing entropy. This is the difference between building a nuclear bomb and keeping it, as opposed to building the bomb to use it. When one was used, it was meant to end a war, not start one. And success in life may as well be defined by hoarding order, be it technologically, financially, socially, or just objects. The pyramids were a feat in lowering entropy, not increasing it. And we love our diamonds. This is also an extrapolation from the evidence in biology that energy entering a system increases order and contributes to the orderly structuring of matter and hence life [1]. [1] https://www.quantamagazine.org/a-new-thermodynamics-theory-of-the-origin-of-life-20140122/ https://www.quantamagazine.org/a-new-thermodynamics-theory-o...
SeanFerree 6y ago
Awesome article! Thanks for sharing!
johnsmith4739 6y ago
"several studies have shown that individuals demonstrate a preference for choice, or the availability of multiple options, over and above utilitarian value." -> yes, it is called the need for orientation/control and "utilitarian value" has nothing to do with it -> Index Funds vs. Actively-Managed Funds -> people prefer the latter even if the returns are consistently lower. [1] "Yet we lack a decision-making framework that integrates preference for choice with traditional utility maximisation in free choice behaviour." -> utility maximisation "has charm for economists, but it rests on the shaky foundation of an implausible and untestable assumption" - Daniel Kahneman [2] -> TL;DR the author of "Thinking Fast and Slow" proves it false "We found that participants were biased towards states that kept their options open, even when both states were balanced in the total number of goal locations. This bias was evident not only when both contexts were equally valuable but throughout all value conditions..." AND "Participants were not informed of the precise values ..." -> seeing the utilitarian variable being forced upon conclusions is disheartening [1] https://www.thebalance.com/index-funds-vs-actively-managed-funds-2466445 https://www.thebalance.com/index-funds-vs-actively-managed-f... [2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=870494 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=870494

103 points by Gaessaki 6y ago

There is some work that explores this from an AI perspective with relative success [1]. It turns out that you can create quite intelligent agents if they look to maximize future entropy, with interesting results [2]. It's still quite computationally expensive, but your normal search reduction tricks apply and you can get something computationally feasible. [1] https://arxiv.org/abs/1310.1863 https://arxiv.org/abs/1310.1863 [2] https://www.mdpi.com/1099-4300/16/5/2789 https://www.mdpi.com/1099-4300/16/5/2789

sriku 6y ago
An aside: this is great concept naming - "empowerment" for the channel capacity between sensors and actuation for an agent. It is at once relatable, fairly precise, memorable/recallable and therefore searchable .. instead of "PersonName's capacity" or something like that.
medlyyy 6y ago
That's a really great, interesting paper ([2]). Thanks for linking it here. This is a more recent (Dec 2020) paper by one of the authors on combining empowerment and extrinsic goals: https://ieeexplore.ieee.org/abstract/document/9284556 https://ieeexplore.ieee.org/abstract/document/9284556

PeterisP 6y ago

Isn't it pretty much the optimal behavior as evidenced e.g. by multi-armed bandit algorithms and explore-exploit balance in reinforcement learning?

johbjo 6y ago
Algorithms would try to maximize expected value. As far as I can tell, here it seems humans value choices over expected value. In other words, humans pay for the perception of freedom.
- PeterisP 6y ago
  No, my point is that our research on algorithms dealing with uncertain rewards show that in such scenarios intentionally exploring choices (entropy maximization) is the optimal way to maximize long-term expected value. I.e. it's not that humans value choices over expected value, since valuing choices actually is the correct way to get larger expected value (with caveats such as how explore vs exploit tradeoff needs to be changed over time) - the message isn't that humans "pay for the perception of freedom" but that human evolved values, even seemingly irrational such as "need for perception of freedom" are actually close to mathematically optimal behavior.
  - aaronblohowiak 6y ago
    This seems like another way of stating "paying for optionality under uncertainty is generally worth it", which seems like a well-established result.
    - PeterisP 6y ago
      Well, testing whether the intuitive behavior of humans matches that or not is always an interesting result.

FlyingSaucer 6y ago

This topic is extremely interesting, and good to see that experiments support it. In AI lens: In a way, you can compare this to novelty seeking and intelligent exploration which is quite an active field in Artificial Life and game AI[1]. If you find this interesting: Jeff Clune, Kenneth Stanley and Joel Lehman conducted interesting related research. Also, isn't this somehow related to the Free Energy principle by Karl Friston? If you look at entropy maximization as a way to minimize surprises. [1] : https://arxiv.org/abs/1901.10995 https://arxiv.org/abs/1901.10995

netizen-9748 6y ago
Friston's principle of free energy was actually the first thing that came to mind on reading the title, it's an absolutely fascinating idea.

hkt 6y ago

On an intuitive level, this makes perfect sense. Assuming that entropy is, roughly speaking, novelty, that makes the calculation one about exploring new options for utility gains. I recall seeing a study (although not where) suggesting novelty seeking was a key hallmark of intelligence. Maybe this means the entropy-utility calibration drives their intelligence? (Alongside their actual material circumstances)

alfonsodev 6y ago
It's like the reverse of a boat breaking the ice while navigating the North Pole, I imagine you want to be moving towards entropy and leaving behind the opposite, in the edge between certainty and novelty. But also in loops, because we have to unlearn time to time, and break the ice, ok now I'm lost in my own metaphor :D
- hkt 6y ago
  Lost in your metaphor as you might be, I like the idea of humans as moving towards entropy and leaving behind order..!

_nhynes 6y ago

Of course. That's why the first two versions of the Matrix failed.

johndoe42377 6y ago

There is no such thing as evidence for abstract concepts.

macawfish 6y ago
Prove it
- johndoe42377 6y ago
  Easily. It is a type error. An attempt to apply inapplicable (orthogonal) concepts to one another. An evidence implies an observable phenomenon (whose description is a statement of fact). Merely a chain of reasoning is never an evidence, because it usually based on flawed premises or domain-specific logic. Now pay me.
godelzilla 6y ago
>These results show that free choice behaviour is well explained by an inference-based framework in which both utility and entropy are optimised. Yeah, I LOL'd at this conclusion in the abstract. This is a great example of economics not being science.

nemoniac 6y ago

Interesting paper. The exploration Vs exploitation payoff measured.

dmichulke 6y ago

aka Mobility heuristics A very strong heuristics that works well in many games (exceptions are usually very interesting games) and is the root to other heuristic concepts such as piece value, central positioning, "protected king", ... in Chess and similar concepts in, e.g., Starcraft. Also very easy to implement, for discrete turn-based games it's just the number of moves in a given state. http://ggp.stanford.edu/lectures/heuristics.pdf http://ggp.stanford.edu/lectures/heuristics.pdf

juskrey 6y ago

Look for Kelly criterion for the whole field on how to cook entropy

gglon 6y ago

Also see Casual Entropic Forces [1] by A. D. Wissner-Gross [2] and C. E. Freer [1] https://www.alexwg.org/publications/PhysRevLett_110-168702.pdf https://www.alexwg.org/publications/PhysRevLett_110-168702.p... [2] https://www.alexwg.org/ https://www.alexwg.org/

r-zip 6y ago

Their definition of entropy is missing a negative sign...I understand that this is a preprint but come on.

yottalove 6y ago

When my son has asked me what he should do about courses or employment or even vacation choices, I answer that he ought to choose the path that gives him the most choices. He has thanked me many time for that advice, which has resulted in a high-value path for him.

unabst 6y ago

Are we sure the availability of options equals entropy? It doesn't appear as though we all act to simply increase our options. Preferring options over reward may also constitute delayed gratification and sacrifice, which is another interesting can of worms, but can it be predicated in terms of just preference of options over reward when those are your only two artificial options? Human behavior appears to point towards the maximization of current order as an investment in power/potential to drive future entropy, as opposed to simply maximizing entropy. This is the difference between building a nuclear bomb and keeping it, as opposed to building the bomb to use it. When one was used, it was meant to end a war, not start one. And success in life may as well be defined by hoarding order, be it technologically, financially, socially, or just objects. The pyramids were a feat in lowering entropy, not increasing it. And we love our diamonds. This is also an extrapolation from the evidence in biology that energy entering a system increases order and contributes to the orderly structuring of matter and hence life [1]. [1] https://www.quantamagazine.org/a-new-thermodynamics-theory-of-the-origin-of-life-20140122/ https://www.quantamagazine.org/a-new-thermodynamics-theory-o...

SeanFerree 6y ago

Awesome article! Thanks for sharing!

johnsmith4739 6y ago

"several studies have shown that individuals demonstrate a preference for choice, or the availability of multiple options, over and above utilitarian value." -> yes, it is called the need for orientation/control and "utilitarian value" has nothing to do with it -> Index Funds vs. Actively-Managed Funds -> people prefer the latter even if the returns are consistently lower. [1] "Yet we lack a decision-making framework that integrates preference for choice with traditional utility maximisation in free choice behaviour." -> utility maximisation "has charm for economists, but it rests on the shaky foundation of an implausible and untestable assumption" - Daniel Kahneman [2] -> TL;DR the author of "Thinking Fast and Slow" proves it false "We found that participants were biased towards states that kept their options open, even when both states were balanced in the total number of goal locations. This bias was evident not only when both contexts were equally valuable but throughout all value conditions..." AND "Participants were not informed of the precise values ..." -> seeing the utilitarian variable being forced upon conclusions is disheartening [1] https://www.thebalance.com/index-funds-vs-actively-managed-funds-2466445 https://www.thebalance.com/index-funds-vs-actively-managed-f... [2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=870494 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=870494