4 ms·

this is an interesting problem but it’s more of a logic problem than a true test of theory of mind. when i think “theory of mind” i think being able to model an

by extr 2y ago

this is an interesting problem but it’s more of a logic problem than a true test of theory of mind. when i think “theory of mind” i think being able to model an external agent with complete knowledge, incentives, and behavior. i would not doubt LLMs have something close to this for humans, almost by accident since they are trained on human outputs.

emseetech 2y ago
I would venture to guess that it has value in that we couldn't even begin to properly test a theory of mind if they can't yet reason through a logic puzzle. It would be like testing if a society could build a rocket ship when they don't know how to use tools.
- extr 2y ago
  Children as young as 4 demonstrate theory of mind but would be unable to solve artificial logic problems like OP
  - emseetech 2y ago
    Fair point, but I'd say that I don't think of LLMs as anthropomorphic individuals and more like societies acting in statistical concert. Maybe this is a failure on my part and is the wrong approach.
- zmgsabst 2y ago
  What animals have theory of mind?
  - sdenton4 2y ago
    Many, certainly many primates. Our ability to understand animal intelligence is limited by or ability to ask appropriate questions, so we tend to underestimate intelligence outside of the primate space. Here's an excellent book on this topic: https://www.goodreads.com/book/show/30231743-are-we-smart-enough-to-know-how-smart-animals-are https://www.goodreads.com/book/show/30231743-are-we-smart-en...
  - aithrowawaycomm 2y ago
    Ravens do: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1559847/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1559847/ I don't have a link but I believe squirrels do, for the exact same reason: their lives are full of intense intra-species competition, with other squirrels constantly raiding each others' stashes. IIRC squirrels will even deceptively hide acorns if they know another squirrel is watching.
nightsd01 2y ago
I think you are right here - the ability to test theory of mind in an LLM would be more like testing how well it can distinguish its own motivations/ideas from that of a separate entity. I would agree that this question is more of a logic puzzle and less of a real test of 'theory of mind' In fact, just to have a theory of mind, it kind of assumes you have a mind, with your own ideas/motivations/etc

by extr 2y ago

emseetech 2y ago

I would venture to guess that it has value in that we couldn't even begin to properly test a theory of mind if they can't yet reason through a logic puzzle. It would be like testing if a society could build a rocket ship when they don't know how to use tools.

extr 2y ago
Children as young as 4 demonstrate theory of mind but would be unable to solve artificial logic problems like OP
- emseetech 2y ago
  Fair point, but I'd say that I don't think of LLMs as anthropomorphic individuals and more like societies acting in statistical concert. Maybe this is a failure on my part and is the wrong approach.
zmgsabst 2y ago
What animals have theory of mind?
- sdenton4 2y ago
  Many, certainly many primates. Our ability to understand animal intelligence is limited by or ability to ask appropriate questions, so we tend to underestimate intelligence outside of the primate space. Here's an excellent book on this topic: https://www.goodreads.com/book/show/30231743-are-we-smart-enough-to-know-how-smart-animals-are https://www.goodreads.com/book/show/30231743-are-we-smart-en...
- aithrowawaycomm 2y ago
  Ravens do: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1559847/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1559847/ I don't have a link but I believe squirrels do, for the exact same reason: their lives are full of intense intra-species competition, with other squirrels constantly raiding each others' stashes. IIRC squirrels will even deceptively hide acorns if they know another squirrel is watching.

nightsd01 2y ago

I think you are right here - the ability to test theory of mind in an LLM would be more like testing how well it can distinguish its own motivations/ideas from that of a separate entity. I would agree that this question is more of a logic puzzle and less of a real test of 'theory of mind' In fact, just to have a theory of mind, it kind of assumes you have a mind, with your own ideas/motivations/etc