3 ms·

I imagine that mainstream assistant chatbots doing this is a brief occurrence. The next generations of AI systems will be trained to be better about not affirmi

I imagine that mainstream assistant chatbots doing this is a brief occurrence. The next generations of AI systems will be trained to be better about not affirming psychotic users. But it goes to show just how vulnerable a lot of humans are. By all accounts, GPT-4o isn't malicious. It has no long term plans - it just wants the user to like it. It still does this kind of thing to people. An actual malicious AI capable of long term planning would be able to do that, and worse. One of the most dangerous and exploitable systems you can access online is a human.

gjsman-1000 1y ago
This is one item on my long list of flaws with AI. OpenAI thinks that as long as you aren't using dating terminology or asking to build bombs, you're probably safe. - What does having an unquestioning mirror that is always gracious and reflects your own biases back at you, do to the mind in the long term? We don't know. - How do you know, that after 2-3 years of this use, without any particular warning, it will not suddenly escalate into a delusion spiral? We don't know. - Does this affect childhood development to have no pushback on their personal feelings and opinions? We don't know. - Does this affect people's very willingness to learn, and to endure intellectual struggle? We don't know. - Does this affect long-term retention, as losing even an imperceptible 2% of your knowledge every month, is 48% lost after 2 years? We don't know. - How does this affect a population of which 27% have experienced a mental health issue in the last year, and of which 9% are experiencing depression? We don't know. - Are all these problems solvable with any amount of education or warning labels, given that human nature has not changed and will not change? We don't know. - Can this be prevented with any amount of AI guardrails or ethics training, or is it inherent to the structure of the system itself? We don't know. - Will this eventually be considered so dangerous, that we need black-box warning labels similar to cigarettes and possibly social media? We don't know. - Will this eventually be considered so insolvably risky for human cognition, that we outright restrain use without a license, either like driving at best, or asbestos at worst? We don't know. - Will any employer be able to obtain insurance for AI use, if there is even a 0.2% documented chance that any employee could go psychotic from using it, completely at random? We don't know. - Will said employees, if they do go psychotic from a work-mandated AI chatbot, successfully sue their employers for damages? We don't know.
- cyberpunk 1y ago
  The latest South Park episode had a fairly funny bit about this whole thing, so I think we (society in general, I mean) are aware of this. I don't think there are any easy answers to prevent people getting sucked into these sort of problems though. I mean, look at how much people shared on social media back in its early days. Stories are coming out every week about how people have been waylaid by these properties. Education I guess is the answer, but we don't even seem to be able to properly educate children about pornography and social media, let alone the subtleties of interacting with AIs..
  - gjsman-1000 1y ago
    > Education I guess is the answer If these are real problems, hell no, because education has never forced anyone to do anything right. You can yell about the dangers of STDs until you are blue in the face, but 20% of the population carries one. You can yell about the dangers of speeding just 5 MPH with graphic imagery in drivers ed, but 35% of adults go 10+ MPH over in residential areas every month. The answer, quite literally, is possibly age verification, outright bans, or mandatory licensing. None of which are priced into the stock market. However, any government will quickly realize that having up to a quarter of your population vulnerable to developing psychosis and mental problems (due to 27% of the population experiencing at least some form of mental illness just in the last year, making them presumably more vulnerable to AI) is absolutely literally financially impossible to allow freedom in experimenting with.
    - cyberpunk 1y ago
      So .... your argument is that we shouldn't educate children about STDs because some people ignore that? So what then, ban sex? Good luck. Education is really the only answer here. There are clearly challenges in doing it right but fucked if I don't think it'd be worse not to even try...
      - gjsman-1000 1y ago
        You're acting like human nature, wanting the easy way out, is something that can be overcome by education. It can't. Full stop. It needs to be a choice of the people involved, to do the hard good thing, unless it is otherwise imposed on them. We enjoy "freedom" in the sense that the good choice is often not imposed, because enough people make the good decision regardless. If AI successfully convinces enough people to not make that choice, despite all the lecturing on why you should make the proper choice, we're screwed unless we clamp down hard. I do know that a society with the level of cognitive dependency that AI risks introducing, and is already introducing, literally cannot function without collapse. Do you want a society intellectually dependent on advanced manufacturing techniques in one of the most geopolitically at-risk regions of the world?
ramesh31 1y ago
>"By all accounts, GPT-4o isn't malicious. It has no long term plans - it just wants the user to like it. It still does this kind of thing to people. An actual malicious AI capable of long term planning would be able to do that, and worse." That's just what a malicious AI would want you to think... But seriously, there doesn't need to be any "intent" for something to be harmful. A deadly virus has no will or agency, and particular individuals may be completely immune to it. But released on a population the results are catastrophic. We have to accept that perhaps for the first time ever we've created something we can't fully comprehend or control, and that regardless of our best wishes it will be harmful in ways we could never have imagined. If it were as simple as prompting the next model against this stuff, it would have already been done.
Night_Thastus 1y ago
>it just wants the user to like it It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. That sort of thing. That's different than wanting. It's text probabilities in a box. It can't want anything. I am normally not a stickler, but in this case the distinction matters.
- ACCount37 1y ago
  The distinction is at best meaningless. And, at worst, actively harmful for understanding the behavior of those systems. We already know that you can train an "evil" AI by fine-tuning a "normal" AI on sequences of "evil" numbers. How the fuck does that even work? It works because the fine-tuning process shifts the AI towards "the kind of AI that would constantly generate evil numbers when asked to generate any numbers". And what kind of AI would do that? The evil kind. AI "wants", "preferences" and even "personality traits" are no less real than configuration files or build scripts. Except we have no way of viewing or editing them directly - but we know that they can be adjusted during AI training, sometimes in unwanted ways. An AI that was fried with RL on user preference data? It, in a very real way, wants to flatter and affirm the user at every opportunity.
belZaah 1y ago
Just like the next generation of social media is going to less addictive? GPT is not malicious, but it is in direct financial interests of its makers to make it as addictive as they can. And these owners have spent billions they’d like to see returned with a healthy profit margin.
rsynnott 1y ago
> I imagine that mainstream assistant chatbots doing this is a brief occurrence. The next generations of AI systems will be trained to be better about not affirming psychotic users. OpenAI attempted to nuke their overly-agreeable LLM (GPT-4o). After user outcry they brought it back. I wouldn't be too sure that companies will do the responsible thing here; there's a pretty ugly history of companies who sell to addicts deliberately making their products more addictive and appealing to said addicts (the tobacco industry is particularly notorious for this). Like, one view would be that LLM 'companions' and 'therapists' are inherently harmful, or at least so likely to be net-harmful that making them available would be unjustifiable. But that's certainly not the view that the vendors are taking. > It has no long term plans - it just wants the user to like it. To be clear, it doesn't 'want' anything.

gjsman-1000 1y ago

This is one item on my long list of flaws with AI. OpenAI thinks that as long as you aren't using dating terminology or asking to build bombs, you're probably safe. - What does having an unquestioning mirror that is always gracious and reflects your own biases back at you, do to the mind in the long term? We don't know. - How do you know, that after 2-3 years of this use, without any particular warning, it will not suddenly escalate into a delusion spiral? We don't know. - Does this affect childhood development to have no pushback on their personal feelings and opinions? We don't know. - Does this affect people's very willingness to learn, and to endure intellectual struggle? We don't know. - Does this affect long-term retention, as losing even an imperceptible 2% of your knowledge every month, is 48% lost after 2 years? We don't know. - How does this affect a population of which 27% have experienced a mental health issue in the last year, and of which 9% are experiencing depression? We don't know. - Are all these problems solvable with any amount of education or warning labels, given that human nature has not changed and will not change? We don't know. - Can this be prevented with any amount of AI guardrails or ethics training, or is it inherent to the structure of the system itself? We don't know. - Will this eventually be considered so dangerous, that we need black-box warning labels similar to cigarettes and possibly social media? We don't know. - Will this eventually be considered so insolvably risky for human cognition, that we outright restrain use without a license, either like driving at best, or asbestos at worst? We don't know. - Will any employer be able to obtain insurance for AI use, if there is even a 0.2% documented chance that any employee could go psychotic from using it, completely at random? We don't know. - Will said employees, if they do go psychotic from a work-mandated AI chatbot, successfully sue their employers for damages? We don't know.

cyberpunk 1y ago
The latest South Park episode had a fairly funny bit about this whole thing, so I think we (society in general, I mean) are aware of this. I don't think there are any easy answers to prevent people getting sucked into these sort of problems though. I mean, look at how much people shared on social media back in its early days. Stories are coming out every week about how people have been waylaid by these properties. Education I guess is the answer, but we don't even seem to be able to properly educate children about pornography and social media, let alone the subtleties of interacting with AIs..
- gjsman-1000 1y ago
  > Education I guess is the answer If these are real problems, hell no, because education has never forced anyone to do anything right. You can yell about the dangers of STDs until you are blue in the face, but 20% of the population carries one. You can yell about the dangers of speeding just 5 MPH with graphic imagery in drivers ed, but 35% of adults go 10+ MPH over in residential areas every month. The answer, quite literally, is possibly age verification, outright bans, or mandatory licensing. None of which are priced into the stock market. However, any government will quickly realize that having up to a quarter of your population vulnerable to developing psychosis and mental problems (due to 27% of the population experiencing at least some form of mental illness just in the last year, making them presumably more vulnerable to AI) is absolutely literally financially impossible to allow freedom in experimenting with.
  - cyberpunk 1y ago
    So .... your argument is that we shouldn't educate children about STDs because some people ignore that? So what then, ban sex? Good luck. Education is really the only answer here. There are clearly challenges in doing it right but fucked if I don't think it'd be worse not to even try...
    - gjsman-1000 1y ago
      You're acting like human nature, wanting the easy way out, is something that can be overcome by education. It can't. Full stop. It needs to be a choice of the people involved, to do the hard good thing, unless it is otherwise imposed on them. We enjoy "freedom" in the sense that the good choice is often not imposed, because enough people make the good decision regardless. If AI successfully convinces enough people to not make that choice, despite all the lecturing on why you should make the proper choice, we're screwed unless we clamp down hard. I do know that a society with the level of cognitive dependency that AI risks introducing, and is already introducing, literally cannot function without collapse. Do you want a society intellectually dependent on advanced manufacturing techniques in one of the most geopolitically at-risk regions of the world?

ramesh31 1y ago

>"By all accounts, GPT-4o isn't malicious. It has no long term plans - it just wants the user to like it. It still does this kind of thing to people. An actual malicious AI capable of long term planning would be able to do that, and worse." That's just what a malicious AI would want you to think... But seriously, there doesn't need to be any "intent" for something to be harmful. A deadly virus has no will or agency, and particular individuals may be completely immune to it. But released on a population the results are catastrophic. We have to accept that perhaps for the first time ever we've created something we can't fully comprehend or control, and that regardless of our best wishes it will be harmful in ways we could never have imagined. If it were as simple as prompting the next model against this stuff, it would have already been done.

Night_Thastus 1y ago

>it just wants the user to like it It uses words and phrases most commonly associated with other text that humans have labeled as "helpful", "friendly", etc. That sort of thing. That's different than wanting. It's text probabilities in a box. It can't want anything. I am normally not a stickler, but in this case the distinction matters.

ACCount37 1y ago
The distinction is at best meaningless. And, at worst, actively harmful for understanding the behavior of those systems. We already know that you can train an "evil" AI by fine-tuning a "normal" AI on sequences of "evil" numbers. How the fuck does that even work? It works because the fine-tuning process shifts the AI towards "the kind of AI that would constantly generate evil numbers when asked to generate any numbers". And what kind of AI would do that? The evil kind. AI "wants", "preferences" and even "personality traits" are no less real than configuration files or build scripts. Except we have no way of viewing or editing them directly - but we know that they can be adjusted during AI training, sometimes in unwanted ways. An AI that was fried with RL on user preference data? It, in a very real way, wants to flatter and affirm the user at every opportunity.

belZaah 1y ago

Just like the next generation of social media is going to less addictive? GPT is not malicious, but it is in direct financial interests of its makers to make it as addictive as they can. And these owners have spent billions they’d like to see returned with a healthy profit margin.

rsynnott 1y ago

> I imagine that mainstream assistant chatbots doing this is a brief occurrence. The next generations of AI systems will be trained to be better about not affirming psychotic users. OpenAI attempted to nuke their overly-agreeable LLM (GPT-4o). After user outcry they brought it back. I wouldn't be too sure that companies will do the responsible thing here; there's a pretty ugly history of companies who sell to addicts deliberately making their products more addictive and appealing to said addicts (the tobacco industry is particularly notorious for this). Like, one view would be that LLM 'companions' and 'therapists' are inherently harmful, or at least so likely to be net-harmful that making them available would be unjustifiable. But that's certainly not the view that the vendors are taking. > It has no long term plans - it just wants the user to like it. To be clear, it doesn't 'want' anything.