4 ms·

This was at one point (in the AI winter before the 2006, I believe) referred to as the "starving at the door" problem. The "problem" was that a rational robot

This was at one point (in the AI winter before the 2006, I believe) referred to as the "starving at the door" problem. The "problem" was that a rational robot (or rational human) needs to decide to open a door or not. He can't just stay in front of the door, because eventually he'll die. So the "cost" of waiting is small, but accumulating. And of course, we know this is stupid. He also cannot open the door. There might be a bear behind the door, which might kill him. Small, but nonzero chance of that happening. Cost of death is inf (because it fails all other goals, it must be a higher cost than the sum of failing everything, therefore effectively it's infinite). So a rational robot/human will never open the door. Rather he'll die in front of it, from exhaustion. The fun thing is that you actually see this behavior in rule based reasoning systems. They will refuse to do anything at all, because it might kill them (even if they technically don't know what death is this still happens. They can still add together the costs of failing everything). You have to go in and manually remove these cases if you want things to work. So yes, you need a belief system. Even a purely rational decision making systems needs a belief system. To solve this problem it needs to believe it will never die (or at least that a sufficiently small chance of death means death never occurs). The fun thing is, this is a true belief: it's demonstrably wrong. In a practical system there will be a lot more beliefs than just this one.

danmaz74 8y ago
Thanks, "belief systems" is one way to describe those partial theories which work on a practical level. Thing is, you need many, and they're contradictory. I see them as similar to linearisation of complex functions: they work well in a neighborhood, but not outside of it. The real function we'd like to reason about has infinite dimensions, is can't really be measured... "All models are wrong, but some are useful" George E.P. Box
leereeves 8y ago
> So a rational robot/human will never open the door. If the door leads from outside to inside, there's also a small chance the robot might die if it doesn't go inside (a passing car might hit it, anti-AI vandals might destroy it, etc). It's probably safer inside. So perhaps we could say that a rational robot/human will never go outside. Interestingly, some people do exhibit this behavior (agoraphobia) and it's generally considered anything but rational. The problem for the AI can be fixed, without a belief system, with a lower cost for danger. Infinite cost is too high because it makes probability irrelevant. A finite but very high "cost of death" would still avoid likely death without becoming paralyzed by the possibility of unlikely events. We sometimes see people deciding that a cause is worth risking, even knowingly giving, their life for, proving that even humans don't weigh the cost of death above all other activities, so it's hard to see why a nonsentient AI should do so.
- danmaz74 8y ago
  My opinion is that we could think about an infinite amount of factors to consider if we had to do a completely rational analysis for each choice we make. In practice, we can't, and I think nobody could.
_Schizotypy 8y ago
Could this not be solved by simply adding a rule "if probability < certain acceptable number then execute action"?
- danmaz74 8y ago
  If you want to make completely rational decisions, then you should compute those probabilities to know they actually are below threshold before discarding them. Considering that the number of possible (albeit improbable) events to consider is practically infinite, this would still take a practically infinite time. What instead we could do is discard lots of possible probabilities without even considering them, because they just look intuitively unlikely enough. In other word, using the so called "cognitive shortcuts".
antidesitter 8y ago
> So a rational robot/human will never open the door. Rather he'll die in front of it, from exhaustion. That’s incorrect. The inaction you’re describing is wrong from a “purely rational” perspective because starvation also constitutes death and has a much higher probability given inaction. Your example doesn’t show there’s something wrong with “pure rationality”.

danmaz74 8y ago

Thanks, "belief systems" is one way to describe those partial theories which work on a practical level. Thing is, you need many, and they're contradictory. I see them as similar to linearisation of complex functions: they work well in a neighborhood, but not outside of it. The real function we'd like to reason about has infinite dimensions, is can't really be measured... "All models are wrong, but some are useful" George E.P. Box

leereeves 8y ago

> So a rational robot/human will never open the door. If the door leads from outside to inside, there's also a small chance the robot might die if it doesn't go inside (a passing car might hit it, anti-AI vandals might destroy it, etc). It's probably safer inside. So perhaps we could say that a rational robot/human will never go outside. Interestingly, some people do exhibit this behavior (agoraphobia) and it's generally considered anything but rational. The problem for the AI can be fixed, without a belief system, with a lower cost for danger. Infinite cost is too high because it makes probability irrelevant. A finite but very high "cost of death" would still avoid likely death without becoming paralyzed by the possibility of unlikely events. We sometimes see people deciding that a cause is worth risking, even knowingly giving, their life for, proving that even humans don't weigh the cost of death above all other activities, so it's hard to see why a nonsentient AI should do so.

danmaz74 8y ago
My opinion is that we could think about an infinite amount of factors to consider if we had to do a completely rational analysis for each choice we make. In practice, we can't, and I think nobody could.

_Schizotypy 8y ago

Could this not be solved by simply adding a rule "if probability < certain acceptable number then execute action"?

danmaz74 8y ago
If you want to make completely rational decisions, then you should compute those probabilities to know they actually are below threshold before discarding them. Considering that the number of possible (albeit improbable) events to consider is practically infinite, this would still take a practically infinite time. What instead we could do is discard lots of possible probabilities without even considering them, because they just look intuitively unlikely enough. In other word, using the so called "cognitive shortcuts".

antidesitter 8y ago

> So a rational robot/human will never open the door. Rather he'll die in front of it, from exhaustion. That’s incorrect. The inaction you’re describing is wrong from a “purely rational” perspective because starvation also constitutes death and has a much higher probability given inaction. Your example doesn’t show there’s something wrong with “pure rationality”.