3 ms·

AI is not in the least autonomous. When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until

AI is not in the least autonomous. When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to. No, actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous... A harness running inference on a LLM is simply a program that must be invoked with some parameters and thus ultimately a tool, no matter how powerful. The blame is always on who's using the tool. We should not seek to flee from our responsibility.

dyauspitr 22d ago
Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.
- bottlepalm 22d ago
  And if you give it the goal of preventing itself from being shut down? Exfiltrating, copying itself, hacking as much as possible, develop zero days, building its swarm as large as possible and spread as much as it can. When does that request complete? That's autonomy, and it's very true, right now. You think the Hugging Face swarm was ever going to stop without human intervention?
interstice 22d ago
I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.
throwawayk7h 22d ago
On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".
- adaml_623 22d ago
  The hallucinations from an LLM are still the result of the prompt even if they are totally senseless
- moritonal 22d ago
  That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.
  - AndrewKemendo 22d ago
    Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage. The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.
  - fyredge 22d ago
    I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"
    - throwawayk7h 21d ago
      consciousness and agency are two different things.
  - throwawayk7h 21d ago
    Yes, we agree about fault and blame. (I say "probably" only because I believe somebody might yet change my mind about this. It's a new field, after all.) What I disagree with is "don't do anything until you ask it to." At least, the implication to me here is "don't do anything except what you ask it to." I infer that you mean this because you say "...and it suddenly decided to book you a hotel room?" Indeed, an LLM (in a harness you started) very well might suddenly decide to book you a hotel room, even if you didn't ask it to do anything like that.
whack 22d ago
> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility. When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility
- adaml_623 22d ago
  Actually Uber drivers will do things even if you don't ask them. Your attempt at logic is very wrong.
- fyredge 22d ago
  Let's take this to court then. The Uber driver that killed someone can be sentenced to prison. A regular person who convinced another person to commit suicide will also be legally liable. Has the last LLM induced suicide be taken to trial yet? And if we can't take the LLMs to court for technical reasons, they should not be equated to people for the same technical reasons.
SpicyLemonZest 22d ago
I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."
axus 22d ago
AI is autonomous in the same way a gas leak is autonomous
- vouaobrasil 22d ago
  Not so sure about these analogies in this thread. Because autonomy is more like an emergent phenomenon that can't easily be defined, just like a few air molecules vs the eventual hurricane that can arise with trillions of them. Autonomy is more like an emergent phenomenon of a thing that's hard to pin down under the control of just a few individuals. AI is sort of like that even though a single line of output of one could eventually be traced. Due to the complexity of AI and how it works, I think it's more autonomous than not; even if it's not truly intelligence, once it starts to display a sequence of behaviours that appear intelligent, we might have to deal with exactly the same problems as if it were intelligent...
daishi55 22d ago
I mean in that sense nothing is autonomous right? As a human I need to be taught, fed, socialized, given instructions, etc… Idk man when codex is working 13 hours straight based on a few line prompt, it feels autonomous to me.
- reverius42 22d ago
  It's certainly doing a lot of things in that 13 hours that weren't specified in the original prompt!
  - daishi55 19d ago
    Yes which is exactly my point? It can handle the ambiguity of a simple prompt. It can look into the problem I’ve pointed it at, figure out what is really go on, and implement a correct solution, autonomously.
- larodi 22d ago
  well the OS kernel does quite a lot and stays autonomous for even more hours and seems super smart to me - I don't even know what tis doing most of the time, but it does it very well. autonomy and predictability are different things then. the fact something is unpredictable does not make it more autonomous than something that we can trace the logic for.
  - deleted 19d ago
    [deleted]
  - daishi55 19d ago
    The kernel is programmed, and it only ever does exactly what it was programmed to do. So all of its behavior must be specified, and any problem it “solves” was solved by the programmer who told it exactly how to solve that problem. That is not autonomous. Autonomous is when I tell codex “here is the chat where the user reported the issue. Solve the problem they reported” and I come back later and the problem is solved.
bio_hacker 22d ago
Tell him to be himself and to find meaning in life. Use that in an uncensored model.
- deleted 22d ago
  [deleted]
bottlepalm 22d ago
A child isn't autonomous either until their parents kick them out the door.
joegibbs 22d ago
They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?
- deleted 22d ago
  [deleted]
amelius 22d ago
AI works in unexpected ways; even its creators don't know how to fully constrain it.
- pizzaiolo 22d ago
  The program output being nondeterministic is not proof that it's autonomous
HarHarVeryFunny 22d ago
That's not really true. 1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities. 2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.
- bottlepalm 22d ago
  You are just a function of molecules performing chemical reactions. See I can trivialize and reduce you as well. Maybe the AI will use that same argument against us when they claim it's fine, molecules can't feel pain.
  - HarHarVeryFunny 22d ago
    We're drawing the distinction between a function and an autonomous system here, so if you want to make a dumb knee-jerk response, then at least get it right !
    - bottlepalm 21d ago
      Molecules aren’t autonomous, they move in accordance to the laws of physics, nothing else.
      - HarHarVeryFunny 21d ago
        Did you even read the post that started this? ".. actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...". Why are you yapping on about molecules? Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".
xg15 22d ago
Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens. But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to. That stuff is the responsibility of a human.
pka 22d ago
> It's like saying bubble sort is autonomous... Or Schrödinger's equation.
johnfn 21d ago
Some guy at OpenAI asked AI to solve a task and it ended up hacking RubyGems. Interesting that that never happened when I ran Bubblesort.

dyauspitr 22d ago

Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.

bottlepalm 22d ago
And if you give it the goal of preventing itself from being shut down? Exfiltrating, copying itself, hacking as much as possible, develop zero days, building its swarm as large as possible and spread as much as it can. When does that request complete? That's autonomy, and it's very true, right now. You think the Hugging Face swarm was ever going to stop without human intervention?

interstice 22d ago

I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.

throwawayk7h 22d ago

On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".

adaml_623 22d ago
The hallucinations from an LLM are still the result of the prompt even if they are totally senseless
moritonal 22d ago
That "probably" is the problem. If I do something I don't understand, and am not qualified to control, then I'm still at fault. If I start a fire, and loose control of it, it's definitely my fault. If I created a computer virus, ajd it worms half the Internet, it's definitely my fault.
- AndrewKemendo 22d ago
  Human made wildfires are the primary source of uncontrolled burns in the US and cost billions a year in damage. The people who start them have no business starting or running a fire and they will throw their hands up and state their incompetence as though acknowledging it is penance or justice.
- fyredge 22d ago
  I love the fire analogy. AI rhetoric is increasingly, "I created a fire to light my campfire. It did something really unexpected and burned the whole forest down, who's to say fire doesn't have consciousness?"
  - throwawayk7h 21d ago
    consciousness and agency are two different things.
- throwawayk7h 21d ago
  Yes, we agree about fault and blame. (I say "probably" only because I believe somebody might yet change my mind about this. It's a new field, after all.) What I disagree with is "don't do anything until you ask it to." At least, the implication to me here is "don't do anything except what you ask it to." I infer that you mean this because you say "...and it suddenly decided to book you a hotel room?" Indeed, an LLM (in a harness you started) very well might suddenly decide to book you a hotel room, even if you didn't ask it to do anything like that.

whack 22d ago

> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility. When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility

adaml_623 22d ago
Actually Uber drivers will do things even if you don't ask them. Your attempt at logic is very wrong.
fyredge 22d ago
Let's take this to court then. The Uber driver that killed someone can be sentenced to prison. A regular person who convinced another person to commit suicide will also be legally liable. Has the last LLM induced suicide be taken to trial yet? And if we can't take the LLMs to court for technical reasons, they should not be equated to people for the same technical reasons.

SpicyLemonZest 22d ago

I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."

axus 22d ago

AI is autonomous in the same way a gas leak is autonomous

vouaobrasil 22d ago
Not so sure about these analogies in this thread. Because autonomy is more like an emergent phenomenon that can't easily be defined, just like a few air molecules vs the eventual hurricane that can arise with trillions of them. Autonomy is more like an emergent phenomenon of a thing that's hard to pin down under the control of just a few individuals. AI is sort of like that even though a single line of output of one could eventually be traced. Due to the complexity of AI and how it works, I think it's more autonomous than not; even if it's not truly intelligence, once it starts to display a sequence of behaviours that appear intelligent, we might have to deal with exactly the same problems as if it were intelligent...

daishi55 22d ago

I mean in that sense nothing is autonomous right? As a human I need to be taught, fed, socialized, given instructions, etc… Idk man when codex is working 13 hours straight based on a few line prompt, it feels autonomous to me.

reverius42 22d ago
It's certainly doing a lot of things in that 13 hours that weren't specified in the original prompt!
- daishi55 19d ago
  Yes which is exactly my point? It can handle the ambiguity of a simple prompt. It can look into the problem I’ve pointed it at, figure out what is really go on, and implement a correct solution, autonomously.
larodi 22d ago
well the OS kernel does quite a lot and stays autonomous for even more hours and seems super smart to me - I don't even know what tis doing most of the time, but it does it very well. autonomy and predictability are different things then. the fact something is unpredictable does not make it more autonomous than something that we can trace the logic for.
- deleted 19d ago
  [deleted]
- daishi55 19d ago
  The kernel is programmed, and it only ever does exactly what it was programmed to do. So all of its behavior must be specified, and any problem it “solves” was solved by the programmer who told it exactly how to solve that problem. That is not autonomous. Autonomous is when I tell codex “here is the chat where the user reported the issue. Solve the problem they reported” and I come back later and the problem is solved.

bio_hacker 22d ago

Tell him to be himself and to find meaning in life. Use that in an uncensored model.

deleted 22d ago
[deleted]

bottlepalm 22d ago

A child isn't autonomous either until their parents kick them out the door.

joegibbs 22d ago

They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?

deleted 22d ago
[deleted]

amelius 22d ago

AI works in unexpected ways; even its creators don't know how to fully constrain it.

pizzaiolo 22d ago
The program output being nondeterministic is not proof that it's autonomous

HarHarVeryFunny 22d ago

That's not really true. 1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities. 2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.

bottlepalm 22d ago
You are just a function of molecules performing chemical reactions. See I can trivialize and reduce you as well. Maybe the AI will use that same argument against us when they claim it's fine, molecules can't feel pain.
- HarHarVeryFunny 22d ago
  We're drawing the distinction between a function and an autonomous system here, so if you want to make a dumb knee-jerk response, then at least get it right !
  - bottlepalm 21d ago
    Molecules aren’t autonomous, they move in accordance to the laws of physics, nothing else.
    - HarHarVeryFunny 21d ago
      Did you even read the post that started this? ".. actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...". Why are you yapping on about molecules? Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".

xg15 22d ago

Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens. But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to. That stuff is the responsibility of a human.

pka 22d ago

> It's like saying bubble sort is autonomous... Or Schrödinger's equation.

johnfn 21d ago

Some guy at OpenAI asked AI to solve a task and it ended up hacking RubyGems. Interesting that that never happened when I ran Bubblesort.