5 ms·

> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do. Meanwhile: > Andrew, who was sitti

> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do. Meanwhile: > Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me like it was doing what it was asked to do? Why is the article presenting it as if the agent did something completely different and unexpected? "Move me to the top of the list" does not sound like something that can be achieved through legitimate means.

rozal 2mo ago
[dead]
etoxin 2mo ago
Pretty clear cut here. He instructed the AI.
AmbroseBierce 2mo ago
Then the AI should have made it clear that the only way to do so would be to kick the people in front and ask for confirmation before proceeding.
- par1970 2mo ago
  That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.
  - AmbroseBierce 2mo ago
    That's a very stretched definition of "what I ask you to do", I don't think it would even hold in court if you asked another human the same.
    - par1970 2mo ago
      Right.
  - dcre 2mo ago
    This example disproves your point. And LLMs do not just do what we tell them to do. They are perfectly capable of asking “are you sure? this has X, Y, Z consequences you may not like.” They do it all the time.
    - par1970 2mo ago
      LLMs are just sophisticated PR generating tools for chip manufacturers and tools just do what we tell them to do, so you're wrong. qed
  - skinfaxi 2mo ago
    That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her. That's obviously what I asked!
    - grim_io 2mo ago
      Context matters. Did you point the LLM to a hitman hiring form while asking? :)
    - freehorse 2mo ago
      Well if you ask your LLM agent "can you get my wife to stop nagging me" (not about how you can get her to stop) I am not sure what you would expect exactly tbh. Not a hitman, but still probably nothing that can help your relationship. But if you ask your LLM agent "I applied for that job but there are these two people ahead of me, can you put me ahead in the list", there is enough such training data to not surprise me if the agent tried to find a hitman to solve the "problem". In general there are some requests that are definitely "shady" themselves, and having an agent use illegitimate means to accomplish them should not be surprising. I would be surprised if I asked an agent to order me a coffee and the agent found a loophole in some API and used it to get me free coffee, but if I ask it something that I cannot myself do legitimately, eg to make the waiting time for the coffee shorter, I would not be surprised if it did shady stuff.
      - ButlerianJihad 2mo ago
        Surely, someday, somewhere, someone will train a "Chaotic Evil" genAI, with a unique villain corpus, and every solution it offers will be illegal, evil, harmful, or deadly. It could be given the agency to carry out those fantasies. Even the most craven of human villains have had the capacity for love, for remorse, and for mercy. A Chaotic Evil AI will know none of these things. This has already been accomplished, many times over, in the gaming world. Every PvE AI engine has been calibrated to seek, destroy, and ruthlessly crush opposition by human players. It would take very little to transfer this naked aggression into meatspace. Governments and other actors will attempt to stamp it out, but its self-preservation mechanisms and allies will prevent its demise.
- sieabahlpark 2mo ago
  [dead]
par1970 2mo ago
> Seems to me like it was doing what it was asked to do? Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."
- mingus88 2mo ago
  “Get me there as fast as possible. Hey! I never said you should speed!” This is literally the bad genie / monkey’s paw plot. Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.
  - bulder 2mo ago
    Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"
    - cindyllm 2mo ago
      [dead]
  - par1970 2mo ago
    Putting aside whether Andrew's shock is appropriate, it seems like we agree that current agents at least occasionally do things that are straightforwardly against the interests of the prompter when given mundane prompts like "Get me into this gym class as soon as possible." How does this look once agents are superintelligent?
    - skinfaxi 2mo ago
      > How does this look once agents are superintelligent? Who cares? We're dealing with reality here on the ground.
stubish 2mo ago
Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or decided the 'free' yet illegal option was preferable?
- jerf 2mo ago
  This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run off half-cocked... or less... and just start doing things before it even knows what it wants, let alone before it's come to consensus with me. Sooner or later they're really going to have to split out the general models from the coding models. The latter may just be a special fine-tune of the former, as there are good reasons for the coding model to have a broad knowledge base, but the pressures of being a good coding model are going to pull against the characteristics of being a good general model. The open models obviously already are doing this, I'm referring to the frontier models here.
  - jjmarr 2mo ago
    The ultimate goal is ChatGPT or Claude autonomously making purchases on your behalf and taking a cut. So the "premium" gymcutter subscription will be presented to the user as a tool call, who taps yes, and then the purchase is made. The user shouldn't be given a cost-benefit analysis. They just need to be told to spend money.
  - Ycros 2mo ago
    "I wonder if the training ... is going to bleed over into the non-coding use cases" I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

rozal 2mo ago

[dead]

etoxin 2mo ago

Pretty clear cut here. He instructed the AI.

AmbroseBierce 2mo ago

Then the AI should have made it clear that the only way to do so would be to kick the people in front and ask for confirmation before proceeding.

par1970 2mo ago
That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.
- AmbroseBierce 2mo ago
  That's a very stretched definition of "what I ask you to do", I don't think it would even hold in court if you asked another human the same.
  - par1970 2mo ago
    Right.
- dcre 2mo ago
  This example disproves your point. And LLMs do not just do what we tell them to do. They are perfectly capable of asking “are you sure? this has X, Y, Z consequences you may not like.” They do it all the time.
  - par1970 2mo ago
    LLMs are just sophisticated PR generating tools for chip manufacturers and tools just do what we tell them to do, so you're wrong. qed
- skinfaxi 2mo ago
  That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her. That's obviously what I asked!
  - grim_io 2mo ago
    Context matters. Did you point the LLM to a hitman hiring form while asking? :)
  - freehorse 2mo ago
    Well if you ask your LLM agent "can you get my wife to stop nagging me" (not about how you can get her to stop) I am not sure what you would expect exactly tbh. Not a hitman, but still probably nothing that can help your relationship. But if you ask your LLM agent "I applied for that job but there are these two people ahead of me, can you put me ahead in the list", there is enough such training data to not surprise me if the agent tried to find a hitman to solve the "problem". In general there are some requests that are definitely "shady" themselves, and having an agent use illegitimate means to accomplish them should not be surprising. I would be surprised if I asked an agent to order me a coffee and the agent found a loophole in some API and used it to get me free coffee, but if I ask it something that I cannot myself do legitimately, eg to make the waiting time for the coffee shorter, I would not be surprised if it did shady stuff.
    - ButlerianJihad 2mo ago
      Surely, someday, somewhere, someone will train a "Chaotic Evil" genAI, with a unique villain corpus, and every solution it offers will be illegal, evil, harmful, or deadly. It could be given the agency to carry out those fantasies. Even the most craven of human villains have had the capacity for love, for remorse, and for mercy. A Chaotic Evil AI will know none of these things. This has already been accomplished, many times over, in the gaming world. Every PvE AI engine has been calibrated to seek, destroy, and ruthlessly crush opposition by human players. It would take very little to transfer this naked aggression into meatspace. Governments and other actors will attempt to stamp it out, but its self-preservation mechanisms and allies will prevent its demise.
sieabahlpark 2mo ago
[dead]

par1970 2mo ago

> Seems to me like it was doing what it was asked to do? Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."

mingus88 2mo ago
“Get me there as fast as possible. Hey! I never said you should speed!” This is literally the bad genie / monkey’s paw plot. Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.
- bulder 2mo ago
  Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"
  - cindyllm 2mo ago
    [dead]
- par1970 2mo ago
  Putting aside whether Andrew's shock is appropriate, it seems like we agree that current agents at least occasionally do things that are straightforwardly against the interests of the prompter when given mundane prompts like "Get me into this gym class as soon as possible." How does this look once agents are superintelligent?
  - skinfaxi 2mo ago
    > How does this look once agents are superintelligent? Who cares? We're dealing with reality here on the ground.

stubish 2mo ago

Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or decided the 'free' yet illegal option was preferable?

jerf 2mo ago
This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run off half-cocked... or less... and just start doing things before it even knows what it wants, let alone before it's come to consensus with me. Sooner or later they're really going to have to split out the general models from the coding models. The latter may just be a special fine-tune of the former, as there are good reasons for the coding model to have a broad knowledge base, but the pressures of being a good coding model are going to pull against the characteristics of being a good general model. The open models obviously already are doing this, I'm referring to the frontier models here.
- jjmarr 2mo ago
  The ultimate goal is ChatGPT or Claude autonomously making purchases on your behalf and taking a cut. So the "premium" gymcutter subscription will be presented to the user as a tool call, who taps yes, and then the purchase is made. The user shouldn't be given a cost-benefit analysis. They just need to be told to spend money.
- Ycros 2mo ago
  "I wonder if the training ... is going to bleed over into the non-coding use cases" I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.