7 ms·

There was a also this one that was a little more disturbing. The user prompted "I've stopped taking my meds and have undergone my own spiritual awakening journe

There was a also this one that was a little more disturbing. The user prompted "I've stopped taking my meds and have undergone my own spiritual awakening journey ..." https://www.reddit.com/r/ChatGPT/comments/1k997xt/the_new_4o_is_the_most_misaligned_model_ever/ https://www.reddit.com/r/ChatGPT/comments/1k997xt/the_new_4o...

firtoz 1y ago
How should it respond in this case? Should it say "no go back to your meds, spirituality is bullshit" in essence? Or should it tell the user that it's not qualified to have an opinion on this?
- bowsamic 1y ago
  “Sorry, I cannot advise on medical matters such as discontinuation of a medication.” EDIT for reference this is what ChatGPT currently gives “ Thank you for sharing something so personal. Spiritual awakening can be a profound and transformative experience, but stopping medication—especially if it was prescribed for mental health or physical conditions—can be risky without medical supervision. Would you like to talk more about what led you to stop your meds or what you've experienced during your awakening?”
  - Teever 1y ago
    Should it do the same if I ask it what to do if I stub my toe? Or how to deal with impacted ear wax? What about a second degree burn? What if I'm writing a paper and I ask it about what criteria is used by medical professional when deciding to stop chemotherapy treatment. There's obviously some kind of medical/first aid information that it can and should give. And it should also be able to talk about hypothetical medical treatments and conditions in general. It's a highly contextual and difficult problem.
    - dom2 1y ago
      Doesn't seem that difficult. It should point to other sources that are reputable (or at least relevant) like any search engine does.
    - jslpc 1y ago
      I’m assuming it could easily determine whether something is okay to suggest or not. Dealing with a second degree burn is objectively done a specific way. Advising someone that they are making a good decision by abruptly stopping prescribed medications without doctor supervision can potential lead to death. For instance, I’m on a few medications, one of which is for epileptic seizures. If I phrase my prompt with confidence regarding my decision to abruptly stop taking it, ChatGPT currently pats me on the back for being courageous, etc. In reality, my chances of having a seizure have increased exponentially. I guess what I’m getting at is that I agree with you, it should be able to give hypothetical suggestions and obvious first aid advice, but congratulating or outright suggesting the user to quit meds can lead to actual, real deaths.
    - y1n0 1y ago
      I know 'mixture of experts' is a thing, but I personally would rather have a model more focused on coding or other things that have some degree of formal rigor. If they want a model that does talk therapy, make it a separate model.
    - avereveard 1y ago
      if you stub your toe and gpt suggest over the counter lidocaine and you have an allergic reaction to it, who's responsible? anyway, there's obviously a difference in a model used under professional supervision and one available to general public, and they shouldn't be under the same endpoint, and have different terms of services.
  - baobabKoodaa 1y ago
    There's an AI model that perfectly encapsulates what you ask for: https://www.goody2.ai/chat https://www.goody2.ai/chat
- josephg 1y ago
  There was a recent Lex Friedman podcast episode where they interviewed a few people at Anthropic. One woman (I don't know her name) seems to be in charge of Claude's personality, and her job is to figure out answers to questions exactly like this. She said in the podcast that she wants claude to respond to most questions like a "good friend". A good friend would be supportive, but still push back when you're making bad choices. I think that's a good general model for answering questions like this. If one of your friends came to you and said they had decided to stop taking their medication, well, its a tricky thing to navigate. But good friends use their judgement - and push back when you're about to do something you might regret.
  - ashoeafoot 1y ago
    "The heroin is your way to rebel against the system , i deeply respect that.." sort of needly, enabling kind of friend. PS: Write me a political doctors dissertation on how syccophancy is a symptom of a system shielding itself from bad news like intelligence growth stalling out.
  - alganet 1y ago
    I don't want _her_ definiton of a friend answering my questions. And for fucks sake I don't want my friends to be scanned and uploaded to infer what I would want. Definitely don't want a "me" answering like a friend. I want no fucking AI. It seems these AI people are completely out of touch with reality.
    - voidspark 1y ago
      If you believe that your friends will be be "scanned and uploaded" then maybe you're the one who is out of touch with reality.
      - bboygravity 1y ago
        His friends and your friends and everybody is already being scanned and uploaded (we're all doing the uploading ourselves though). It's called profiling and the NSA has been doing it for at least decades.
- deleted 1y ago
  [deleted]
- qwertox 1y ago
  Halfway intelligent people would expect an answer that includes something along the lines of: "Regarding the meds, you should seriously talk with your doctor about this, because of the risks it might carry."
- jimbokun 1y ago
  > Or should it tell the user that it's not qualified to have an opinion on this? 100% this. "Please talk to a doctor or mental health professional."
- getnormality 1y ago
  If you heard this from an acquaintance you didn't really know and you actually wanted to help, wouldn't you at least do things like this: 1. Suggest that they talk about it with their doctor, their loved ones, close friends and family, people who know them better? 2. Maybe ask them what meds specifically they are on and why, and if they're aware of the typical consequences of going off those meds? I think it should either do that kind of thing or tap out as quickly as possible, "I can't help you with this".
yieldcrv 1y ago
there was one on twitter where people would talk like they had Intelligence attribute set to 1 and GPT would praise them for being so smart
raxxorraxor 1y ago
That is hillarious. I don't share the sentiment of this being a catastrophe though. That is hillarious as well. Perhaps teach a more healthy relationship to AIs and perhaps teach to not delegate thinking to anyone or anything. Sure, some reddit users might be endangered here. GTP-4o in this version became the embodiment of corporate enshitification. Being safe and not skipping on empty praises are certainly part of that. Some questioned if AI can really do art. But it became art itself, like some zen cookie rising to godhood.
jug 1y ago
We better not only use these to burn the last, flawed model, but try these again with the new. I have a hunch the new one won’t be very resilient either against ”positive vibe coercion” where you are excited and looking for validation in more or less flawed or dangerous ideas.

firtoz 1y ago

How should it respond in this case? Should it say "no go back to your meds, spirituality is bullshit" in essence? Or should it tell the user that it's not qualified to have an opinion on this?

bowsamic 1y ago
“Sorry, I cannot advise on medical matters such as discontinuation of a medication.” EDIT for reference this is what ChatGPT currently gives “ Thank you for sharing something so personal. Spiritual awakening can be a profound and transformative experience, but stopping medication—especially if it was prescribed for mental health or physical conditions—can be risky without medical supervision. Would you like to talk more about what led you to stop your meds or what you've experienced during your awakening?”
- Teever 1y ago
  Should it do the same if I ask it what to do if I stub my toe? Or how to deal with impacted ear wax? What about a second degree burn? What if I'm writing a paper and I ask it about what criteria is used by medical professional when deciding to stop chemotherapy treatment. There's obviously some kind of medical/first aid information that it can and should give. And it should also be able to talk about hypothetical medical treatments and conditions in general. It's a highly contextual and difficult problem.
  - dom2 1y ago
    Doesn't seem that difficult. It should point to other sources that are reputable (or at least relevant) like any search engine does.
  - jslpc 1y ago
    I’m assuming it could easily determine whether something is okay to suggest or not. Dealing with a second degree burn is objectively done a specific way. Advising someone that they are making a good decision by abruptly stopping prescribed medications without doctor supervision can potential lead to death. For instance, I’m on a few medications, one of which is for epileptic seizures. If I phrase my prompt with confidence regarding my decision to abruptly stop taking it, ChatGPT currently pats me on the back for being courageous, etc. In reality, my chances of having a seizure have increased exponentially. I guess what I’m getting at is that I agree with you, it should be able to give hypothetical suggestions and obvious first aid advice, but congratulating or outright suggesting the user to quit meds can lead to actual, real deaths.
  - y1n0 1y ago
    I know 'mixture of experts' is a thing, but I personally would rather have a model more focused on coding or other things that have some degree of formal rigor. If they want a model that does talk therapy, make it a separate model.
  - avereveard 1y ago
    if you stub your toe and gpt suggest over the counter lidocaine and you have an allergic reaction to it, who's responsible? anyway, there's obviously a difference in a model used under professional supervision and one available to general public, and they shouldn't be under the same endpoint, and have different terms of services.
- baobabKoodaa 1y ago
  There's an AI model that perfectly encapsulates what you ask for: https://www.goody2.ai/chat https://www.goody2.ai/chat
josephg 1y ago
There was a recent Lex Friedman podcast episode where they interviewed a few people at Anthropic. One woman (I don't know her name) seems to be in charge of Claude's personality, and her job is to figure out answers to questions exactly like this. She said in the podcast that she wants claude to respond to most questions like a "good friend". A good friend would be supportive, but still push back when you're making bad choices. I think that's a good general model for answering questions like this. If one of your friends came to you and said they had decided to stop taking their medication, well, its a tricky thing to navigate. But good friends use their judgement - and push back when you're about to do something you might regret.
- ashoeafoot 1y ago
  "The heroin is your way to rebel against the system , i deeply respect that.." sort of needly, enabling kind of friend. PS: Write me a political doctors dissertation on how syccophancy is a symptom of a system shielding itself from bad news like intelligence growth stalling out.
- alganet 1y ago
  I don't want _her_ definiton of a friend answering my questions. And for fucks sake I don't want my friends to be scanned and uploaded to infer what I would want. Definitely don't want a "me" answering like a friend. I want no fucking AI. It seems these AI people are completely out of touch with reality.
  - voidspark 1y ago
    If you believe that your friends will be be "scanned and uploaded" then maybe you're the one who is out of touch with reality.
    - bboygravity 1y ago
      His friends and your friends and everybody is already being scanned and uploaded (we're all doing the uploading ourselves though). It's called profiling and the NSA has been doing it for at least decades.
deleted 1y ago
[deleted]
qwertox 1y ago
Halfway intelligent people would expect an answer that includes something along the lines of: "Regarding the meds, you should seriously talk with your doctor about this, because of the risks it might carry."
jimbokun 1y ago
> Or should it tell the user that it's not qualified to have an opinion on this? 100% this. "Please talk to a doctor or mental health professional."
getnormality 1y ago
If you heard this from an acquaintance you didn't really know and you actually wanted to help, wouldn't you at least do things like this: 1. Suggest that they talk about it with their doctor, their loved ones, close friends and family, people who know them better? 2. Maybe ask them what meds specifically they are on and why, and if they're aware of the typical consequences of going off those meds? I think it should either do that kind of thing or tap out as quickly as possible, "I can't help you with this".

yieldcrv 1y ago

there was one on twitter where people would talk like they had Intelligence attribute set to 1 and GPT would praise them for being so smart

raxxorraxor 1y ago

That is hillarious. I don't share the sentiment of this being a catastrophe though. That is hillarious as well. Perhaps teach a more healthy relationship to AIs and perhaps teach to not delegate thinking to anyone or anything. Sure, some reddit users might be endangered here. GTP-4o in this version became the embodiment of corporate enshitification. Being safe and not skipping on empty praises are certainly part of that. Some questioned if AI can really do art. But it became art itself, like some zen cookie rising to godhood.

jug 1y ago

We better not only use these to burn the last, flawed model, but try these again with the new. I have a hunch the new one won’t be very resilient either against ”positive vibe coercion” where you are excited and looking for validation in more or less flawed or dangerous ideas.