I do not think that resemblance to "Thinking Fast and Slow" is a thing that will necessary happen. Human mind is "designed" that way to overcome some limitation

I do not think that resemblance to "Thinking Fast and Slow" is a thing that will necessary happen. Human mind is "designed" that way to overcome some limitations of his substrate, of a brain made of very slow neurons. AI on the other hand relies on "neurons" that are several orders of magnitude faster. The situations when such an AI could benefit from snap decisions, while having an ability to decide by slow thoughtful process, still could pop, but it will not be like with a human mind. Chess engines have something like that: they can propose a move by a "snap" decision without trying to search all possible developments of a game from a current position. In real games they limit the depth of analysis by time they have. It resembles to some extent "System 1" and "System 2", but there is a gradual transition between them, from depth 0 analysis to depth of infinity. With AI designed for real tasks, not artificial games, it will be probably the same story. They will do a search of depth N in any case, but in situations of time pressure they will keep N very low. Historically attempts to replicate high level understanding of a human mind to build an AI didn't work. Simulations of low level understanding, like neurons and suchlike did work. We can draw parallels between AI developments and human mind traits as we see them, but they are very shaky constructs. At least as shaky as all these psychology "high level" theories, which try to refine naive human understanding how human mind works. > We react to situations and then rationalize why we reacted that way at leisure, and those often turn into excuses and not reasons. I do believe it is not because of limitations of a human mind, but due to training peculiarities. I believe that reasoning itself was "designed" with a goal of communicating inner states of mind to others, by getting an "explanation" that fits the current social situation and helps to reach current social goals. I agree with the idea that politics was a driver of evolution of human intelligence. And it probably still the driver. Humans learned how to apply these new abilities to other kinds of problems, like engineering ones, but it was when their intelligence evolved a lot under a pressure of natural selection driven by politics. Probably people can do better than that, and could understand themselves a way better, could explain their anger by real reasons, but we still learning to seek excuses from the very young age. We are punished when we have "wrong reasons" for our behaviour and reinforced for "good reasons". The idea of such an education is to eliminate wrong reasons and remove their power to influence a person's behaviour, but the unintended consequence is person's preference for excuses over reasons. In other words, it is a cultural thing I believe, not some genetic limitations.