3 ms·

If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make t

If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make typical sociopathic CEOs look like Mister Rogers in comparison. And I'm not sure our legal and social structures have the capacity to absorb that without very very bad things happening.

marsten 7mo ago
Agents playing the iterated prisoner's dilemma learn to cooperate. It's usually not a dominant strategy to be entirely sociopathic when other players are involved.
- ehnto 7mo ago
  You don't get that many iterations in the real world though, and if one of your first iterations is particularly bad you don't get any more iterations.
  - cortesoft 7mo ago
    But AI will train in the artificial world
    - ehnto 7mo ago
      They still fail in the real world, where a single failure can be highly consequential. AI coding is lucky it has early failure modes, pretty low consequence. But I don't see how that looks for an autonomous management agent with arbitrary metrics as goals. Anyone doing AI coding can tell you once an agent gets on the wrong path, it can get very confused and is usually irrecoverable. What does that look like in other contexts? Is restarting the process from scratch even possible in other types of work, or is that unique to only some kinds of work?
  - naasking 7mo ago
    > You don't get that many iterations in the real world though True, for iterations between the same two players, but humans evolved the ability to communicate and so can share the results of past interactions through a network with other agents, aka a reputation. Thus any interaction with a new person doesn't start from a neutral prior.
gotwaz 7mo ago
Not just CEOs, Legal and social structures will also be run by AI. Chimps with 3 inch brains cant handle the level of complexity global systems are currently producing.
AdieuToLogic 7mo ago
> If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make typical sociopathic CEOs look like Mister Rogers in comparison. Algorithms do not possess ethics nor morality[0] and therefore cannot engage in Machiavellianism[1]. At best, algorithms can simulate same as pioneered by ELIZA[2], from which the ELIZA effect[3] could be argued as being one of the best known forms of anthropomorphism. 0 - https://www.psychologytoday.com/us/basics/ethics-and-morality https://www.psychologytoday.com/us/basics/ethics-and-moralit... 1 - https://en.wikipedia.org/wiki/Machiavellianism_(psychology) https://en.wikipedia.org/wiki/Machiavellianism_(psychology) 2 - https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA 3 - https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect
- qsera 7mo ago
  https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect >As Weizenbaum later wrote, "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."... That pretty much explain the AI Hysteria that we observe today.
  - reverius42 7mo ago
    ELIZA couldn't write working code from an English-language prompt though. I think the "AI Hysteria" comes more from current LLMs being actually good at replacing a lot of activity that coders are used to doing regularly. I wonder what Weizenbaum would think of Claude or ChatGPT.
    - qsera 7mo ago
      >ELIZA couldn't write working code from an English-language prompt though. Yea, that is kind of the point. Even such a system could trick people into delusional thinking. > actually good at replacing a lot of activity that coders are used to... I think even that is unrealistic. But that is not what I was thinking. I was thinking when people say that current LLMs will go on improving and reach some kind of real human like intelligence. And ELIZA effect provides a prefect explanation for this. It is very curious that this effect is the perfect thing for scamming investors who are typically bought into such claims, but under ELIZA effect with this, they will do 10x or 100x investment....
tim333 7mo ago
I was kind of worried by them going Machiavellian or evil but it doesn't seem the default state for current ones, I think because they are basically trained on the whole internet which has a lot of be nice type stuff. No doubt some individual humans my try to make them go that way though. I guess it would depend a bit whos interests the AI would be serving. If serving the shareholders it would probably reward creating value for customers, but if it was serving an individual manager competing with others to be CEO say then the optimum strategy might be to go machiavellian on the rivals.
- estearum 7mo ago
  > I think because they are basically trained on the whole internet which has a lot of be nice type stuff. Is this not just because their goals are currently to be seen as "nice"? Surely they can be not-nice if directed to, and then the question is just whether someone can accidentally direct them to do that by e.g. setting up goals that can be more readily achieved by being not-nice. Which... is how many goals in the real world are, which is why the very concept and danger of Machiavellianism exists.
  - tim333 7mo ago
    I've been amused at Musk vs Grok with Grok saying he's the biggest spreader of misinformation and not doing very well when he tells it to go on about white genocide in South Africa. I don't know how easy it is to modify these things in a subtle manner.

marsten 7mo ago

Agents playing the iterated prisoner's dilemma learn to cooperate. It's usually not a dominant strategy to be entirely sociopathic when other players are involved.

ehnto 7mo ago
You don't get that many iterations in the real world though, and if one of your first iterations is particularly bad you don't get any more iterations.
- cortesoft 7mo ago
  But AI will train in the artificial world
  - ehnto 7mo ago
    They still fail in the real world, where a single failure can be highly consequential. AI coding is lucky it has early failure modes, pretty low consequence. But I don't see how that looks for an autonomous management agent with arbitrary metrics as goals. Anyone doing AI coding can tell you once an agent gets on the wrong path, it can get very confused and is usually irrecoverable. What does that look like in other contexts? Is restarting the process from scratch even possible in other types of work, or is that unique to only some kinds of work?
- naasking 7mo ago
  > You don't get that many iterations in the real world though True, for iterations between the same two players, but humans evolved the ability to communicate and so can share the results of past interactions through a network with other agents, aka a reputation. Thus any interaction with a new person doesn't start from a neutral prior.

gotwaz 7mo ago

Not just CEOs, Legal and social structures will also be run by AI. Chimps with 3 inch brains cant handle the level of complexity global systems are currently producing.

AdieuToLogic 7mo ago

> If this was done well in a way that was productive for corporate work, I suspect the AI would engage in Machievelian maneuvering and deception that would make typical sociopathic CEOs look like Mister Rogers in comparison. Algorithms do not possess ethics nor morality[0] and therefore cannot engage in Machiavellianism[1]. At best, algorithms can simulate same as pioneered by ELIZA[2], from which the ELIZA effect[3] could be argued as being one of the best known forms of anthropomorphism. 0 - https://www.psychologytoday.com/us/basics/ethics-and-morality https://www.psychologytoday.com/us/basics/ethics-and-moralit... 1 - https://en.wikipedia.org/wiki/Machiavellianism_(psychology) https://en.wikipedia.org/wiki/Machiavellianism_(psychology) 2 - https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA 3 - https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect

qsera 7mo ago
https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect >As Weizenbaum later wrote, "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."... That pretty much explain the AI Hysteria that we observe today.
- reverius42 7mo ago
  ELIZA couldn't write working code from an English-language prompt though. I think the "AI Hysteria" comes more from current LLMs being actually good at replacing a lot of activity that coders are used to doing regularly. I wonder what Weizenbaum would think of Claude or ChatGPT.
  - qsera 7mo ago
    >ELIZA couldn't write working code from an English-language prompt though. Yea, that is kind of the point. Even such a system could trick people into delusional thinking. > actually good at replacing a lot of activity that coders are used to... I think even that is unrealistic. But that is not what I was thinking. I was thinking when people say that current LLMs will go on improving and reach some kind of real human like intelligence. And ELIZA effect provides a prefect explanation for this. It is very curious that this effect is the perfect thing for scamming investors who are typically bought into such claims, but under ELIZA effect with this, they will do 10x or 100x investment....

tim333 7mo ago

I was kind of worried by them going Machiavellian or evil but it doesn't seem the default state for current ones, I think because they are basically trained on the whole internet which has a lot of be nice type stuff. No doubt some individual humans my try to make them go that way though. I guess it would depend a bit whos interests the AI would be serving. If serving the shareholders it would probably reward creating value for customers, but if it was serving an individual manager competing with others to be CEO say then the optimum strategy might be to go machiavellian on the rivals.

estearum 7mo ago
> I think because they are basically trained on the whole internet which has a lot of be nice type stuff. Is this not just because their goals are currently to be seen as "nice"? Surely they can be not-nice if directed to, and then the question is just whether someone can accidentally direct them to do that by e.g. setting up goals that can be more readily achieved by being not-nice. Which... is how many goals in the real world are, which is why the very concept and danger of Machiavellianism exists.
- tim333 7mo ago
  I've been amused at Musk vs Grok with Grok saying he's the biggest spreader of misinformation and not doing very well when he tells it to go on about white genocide in South Africa. I don't know how easy it is to modify these things in a subtle manner.