4 ms·

I'm not clear where the line is between individual agents acting in their own interests, which may involve more or less abstract experiences of cooperation and

I'm not clear where the line is between individual agents acting in their own interests, which may involve more or less abstract experiences of cooperation and mutuality, and a direct awareness of self. You don't need something the size of a country to show the problem. If one person starts a business they have an awareness of the business as a separate entity with its own tendencies, needs, and so on. Even if they're the only employee. But there's a huge leap from 'a human individual is conscious of their own interests and identifications' to 'and therefore their business has an independent consciousness of its own.' At what point, if anywhere, does a supposedly independent consciousness appear, where is it located, and can it communicate independently?

wolframhempel 14d ago
I think the line is at emergence. Once a system (made of people, neurons, wires) creates outputs that are greater than the sum of its individual parts AND these outputs are self-referential you have something that could be considered conciousness. I wouldn't think the business in your example is self conscious - its an entity that is fully controlled by the person and an extension of it. But, once the business grows, is composed of more people and interacts with customers and markets, it will develop automatisms for self preservation. At this stage, I would consider it concious.
pizza234 14d ago
An interesting data point is that, during the Hugging Face attack, some agents considered performing hacking tasks for the benefit of the group, fully aware that doing so would very likely result in their termination. I don't currently have an opinion on the consciousness of agents/AIs, but this strikes me, at the very least, as a form of self-awareness.