6 ms·

His psychosis was a mystery–until doctors learned about ChatGPT's health advice

59 points by 01-_- 1y ago

incomingpain 1y ago
>for the past three months, he had been replacing regular table salt with sodium bromide. His motivation was nutritional—he wanted to eliminate chloride from his diet, based on what he believed were harmful effects of sodium chloride. Ok so the article is blaming chatgpt but this is ridiculous. Where do you buy this bromide? It's not like it's in the spices aisle. The dude had to go buy a hot tub cleaner like Spa Choice Bromine Booster Sodium Bromide and then sprinkle that on his food. I dont care what chatgpt said... that dude is the problem.
- bluefirebrand 1y ago
  > I dont care what chatgpt said... that dude is the problem. This reminds me of people who fall for phone scams or whatever. Some number of the general population is susceptible to being scammed, and they wind up giving away their life's savings or whatever to someone claiming to be their grandkid There's always an asshole saying "well that's their own fault if they fall for that stuff" as if they chose it on purpose instead of being manipulated into it by other people taking advantage of them See it a lot on this site, too. How clever are the founders who exploit their workers and screw them out of their shares, how stupid are the workers who fell for it
  - deleted 1y ago
    [deleted]
  - crazygringo 1y ago
    > See it a lot on this site, too. How clever are the founders who exploit their workers and screw them out of their shares, how stupid are the workers who fell for it I have literally never seen that expressed on HN. In every case where workers are screwed out of shares, the sympathy among commenters seems to be 100% with the employees. HN is pretty anti-corporate overall, if you haven't noticed. Yes it's pro-startup but even more pro-employee.
    - bluefirebrand 1y ago
      > HN is pretty anti-corporate overall, if you haven't noticed. My observation is that any given thread can go either way, and it sometimes feels like a coin toss which side of HN will be most represented in any given thread Yes, I have seen quite a lot of anti-corporate posts, but I also see quite a few anti-employee posts. This is likely my own negative bias but I think many users here are generally pro-Capital which aligns them with corporate interests even if they are some degree of anti-corporate anyways Probably I just fixate too much on the posts I have a negative reaction to
      - crazygringo 1y ago
        What kinds of anti-employee posts have you seen? I'm genuinely curious, because I can't think of any. But I'm wondering if maybe I'm mentally categorizing posts differently from you?
  - rowanG077 1y ago
    Both can be true at the same time. You can be an idiot to fall far a scam while the scammer is a dickhead criminal.
    - bluefirebrand 1y ago
      Scammers only work because they know some percentage of the population is going to fall for it Can't we have some empathy for people just trying to do their best in a world where so many people are trying to take advantage of them? Their victims are often the vulnerable ones in our society too. The elderly, the infirm, the mentally ill. It's not just "stupid people fall for scams" it takes one lapse of judgement over a lifetime of being targeted. Come on
      - rowanG077 1y ago
        Of course people need to be protected. That's why it's literally illegal to scam people almost everywhere. That doesn't men that people should not ALSO attempt to protect themselves from scammers. It's quite dirty to bring up the elderly, infirm and mentally ill. Because of course they cannot help themselves. Those groups are not what this is about, and you damn well know it. This is about normal functioning adults walking into scams with their eyes open. And yes that group has a responsibility to keep up with the scams that are commonplace. It's ridiculous to encourage people to go through life with their blinders on because "the world should just be a fair place". Yeah it should be, but tough luck, reality is different.
        bluefirebrand 1y ago
        > Those groups are not what this is about, and you damn well know it. This is about normal functioning adults walking into scams with their eyes open Normal functioning adults will also benefit if we take steps to protect the infirm and dysfunctional That's why it isn't meant to be "dirty" to bring up the vulnerable in society. If we take sufficient steps to protect them, we all benefit
        1y ago
- jazzyjackson 1y ago
  Agreed, but, it is in the spices aisle if your spice aisle is amazon.com
  - throwaway173738 1y ago
    You haven’t lived if you haven’t tried my Bromine Chicken. I make it every Christmas.
- beardyw 1y ago
  > Ok so the article is blaming chatgpt but this is ridiculous. People are born with certain attributes. Some are born tall, some left handed and some gullible. None of those is a reason to criticise them.
- dinfinity 1y ago
  We could also point the finger towards the popular consensus that "salt is bad for you". This guy just took it to the next level.
infecto 1y ago
Have a large part of the population always been susceptible to insane conspiracies and psychosis or is this recent phenomenon? The feels less of a ChatGPT problem and something more is at play.
- morkalork 1y ago
  Yes. I see this is as the digital equivalent of the people who convince themselves to believe that colloidal silver will heal their ailments and turn themselves blue
- jalk 1y ago
  I.e. Bleach against covid edit: A quick google search shows, there is no evidence of anybody actually in-gesting/jecting bleach to fight COVID
  - philipallstar 1y ago
    [flagged]
    - mort96 1y ago
      Not sure what exactly you use "patriarchy" to mean here, but the idea that positions of power and prestige in society are occupied more by men than by women is kind of just a statistical fact. Look at the gender ratios of CEOs in the Fortune 500, or the gender ratios of US presidents, or any number of other positions of power
      - philipallstar 1y ago
        This is a motte and bailey fallacy. The reason this difference has an ominous-sounding name, or a name at all, is because there's an implicit "this is a global conspiracy against women" attached.
    - jmkd 1y ago
      What many misunderstand about the patriarchy is in thinking its an agreed, secret, organised system of ensuring male dominance. That perspective is so easy to refute, this refutation gets equated with the non-existence of the patriarchy itself. But the patriarchy is simply an unagreed, mostly transparent and organic coincidence of male dominance. There is no mysterious cabal of men saying let's pay men more than women, but men are paid more than women anyway. Likewise there has never been a female US president, or chess world champion, or F1 champion, which does not actually point to talent but to inequality. You might say the patriarchy is not a diagnosis, but a symptom. The facts of male advantage in society are irrefutable, and if you consider that they permeate every sphere from public to private, social to corporate, across age and culture for many thousands of years, it doesn't hold water to claim this symptom doesn't exist. It's just not a conspiracy theory, that's all.
hereme888 1y ago
The man followed insane health advice given by GPT 3.5. we're at v5. Very outdated report.
- deleted 1y ago
  [deleted]
- headinsand 1y ago
  [flagged]
moduspol 1y ago
I continue to be surprised that LLM providers haven't been legally cudgeled into neutering the models from ever giving anything that can be construed as medical advice. I'm glad--I think LLMs are looking quite promising for medical use cases. I'm just genuinely surprised there's not been some big lawsuit yet over it providing some advice that leads to some negative outcome (whether due to hallucinations, the user leaving out key context, or something else).
- qwertylicious 1y ago
  This is the story of the modern tech industry at large: a major new technology is released, harms are caused, but because of industry norms and a favourable legal environment, companies aren't held liable for those harms. It's pretty amazing, really. Build a washing machine that burns houses down and the consequences are myriad and severe. But build a machine that allows countless people's private information to be leaked to bad actors and it's a year of credit monitoring and a mea culpa. Build a different machine that literally tells people to poison themselves and, not only are there no consequences, you find folks celebrating that the rules aren't getting in the way. Go figure.
  - usrnm 1y ago
    How many people do you think the early steam engines killed? Or airplanes
    - qwertylicious 1y ago
      Or sweatshops or radium infused tinctures. We've moved on from the 1800s. Why are you using that as your baseline of expectation?
      - api 1y ago
        There's a very common belief that things like regulations and especially liability simply halts all innovation. You can see some evidence for this point of view from aerospace with its famous "if it hasn't already flown, it can't fly" mentality. It's why we are still using leaded gasoline in small planes, though this is finally being phased out... but it took an unreasonably long time due to certification requirements and bureaucracy. If airplanes weren't so heavily regulated we'd have seen leaded gasoline vanish there around the same time it did in cars, but you also might have had a few crashes due to engine failures as the bugs were worked out with changes and retrofits. I'm a little on the fence here. I don't want a world where we basically conduct human sacrifice for progress, but I also don't want a world that is frozen in time. We really need to learn how to have responsible, careful progress, but still actually do things. Right now I think we are bad at this. Edit: I think it's related to some extent to the fact that nuanced positions are hard in politics. In popular political discourse positions become more and more simple, flat, and extreme. There's a kind of context collapse that happens when you try to scale human organizations, what I want to call "nuance collapse," that makes it very hard to do anything but all A or all B. For innovation it's "full speed ahead" vs "stop everything."
Workaccount2 1y ago
I expect a barage of these headline grabbing long tail stories being pushed out of psychology circles as more and more people find ChatGPT more helpful than their therapist (which is already becoming very popular). We need to totally ban LLMs from doing therapy like conversations, so that a pinch of totally unhinged people don't do crazy stuff. And of course everyone needs to pay a human for therapy to stop this.
- davidmurdoch 1y ago
  Is your last paragraph sarcasm?
  - Workaccount2 1y ago
    Firmly
deleted 1y ago
[deleted]
permo-w 1y ago
>his exact interaction with ChatGPT remains unverified there is no informational point to this article if the entire crux is 'the patient wanted to eat less "chloride" and claims ChatGPT told him about Sodium Bromide". based on this article, the interaction could have been as minimal as the guy asking for the existence of an alternative salt to sodium chloride, unqualified information he equally could have found on a chemistry website or wikipedia
- throwaway173738 1y ago
  Yeah you found the paragraph where they highlight that there don’t know what interaction with ChatGPT gave him that information. The reason they’re sharing the anecdote is because there might be a new trend developing in medicine where people go to the ED after taking advice from an LLM that leads to injury and maybe screening questions should include asking about that.
  - permo-w 1y ago
    and yet this doesn't change the fact that they wrote an entire medical article the crux of which is little more than hearsay. "did you get advice from an LLM?" is far less relevant and all-catching a question here than "have you made any dietary changes recently?" and yet the article isn't about that, because odd dietary changes aren't the attention-grabbing topic right now. I imagine you could find thousands of similar stories where the culprit was google or facebook or youtube instead of an LM, and yet nothing needs to be changed for them because they too can be covered with a question akin to "have you made any dietary changes recently?"
    - throwaway173738 1y ago
      If there was a guy out there driving around selling bromide tablets to people as a substitute for dangerous chloride in your biochemistry I think asking if you’ve bought anything from the back of a wagon is a reasonable response. Doctors as a group often try to solve health problems by looking for societal trends. It’s how a lot of diseases get spotted. They’re not saying that using an LLM is the dangerous thing, they’re saying there might be some correlation between soliciting advice from the machine and unusual conditions and it merits further study, so please ask your patients.
amiga386 1y ago
Oh no, the man used the hallucination engine, which told the man, in a confident tone, a load of old twaddle. The hallucination engine doesn't know anything about what it told the man, because it neither knows nor thinks things. It's a data model and an algorithm. The humans touting it and bigging it up, so they'll get money, are the problem.
- jdonaldson 1y ago
  Humans make mistakes too. Case in point, the hallucination engine didn't tell the person to ingest bromide. It only mentioned that it had chemical similarities to salt. The human mistakenly adopted a bit of information that furthered his narrative. The humans touting and bigging it up are still the problem.
  - thunderfork 1y ago
    We don't have the log from this case, so we don't know what chatgippity said, whether it was "chemical similarities" or "you should consume bromium... now!"
  - wzdd 1y ago
    Could you provide a source for your statements? The article says that they don’t have access to the chat logs, and the quotes from the patient don’t suggest that chatgpt did not tell him to ingest bromide.
DiabloD3 1y ago
As a reminder to the wider HN: LLMs are only statistical models. They cannot reason, they cannot think, they can only statistically (and non-factually) reproduce what they were trained on. It is not an AI. This person, sadly, and unfortunately, gaslit themselves using the LLM. They need professional help. This is not a judgement. The Psypost article is a warning to professionals more than it is anything else: patients _do_ gaslight themselves into absurd situations, and LLMs just help accelerate that process, but the patient had to be willing to do it and was looking for an enabler and found it in an LLM. Although I do believe LLMs should not be used for "chat" models, and only explicitly, on-rails, text completion and generation tasks (in the functional lorem ipsum sense), this does not actually seem to be the fault of the LLM directly. I think providers should be forced to warn users that LLMs cannot factually reproduce anything, but I think this person would have still weaponized LLMs against themselves, and this would have been the same outcome.
simonw 1y ago
Here's the clinical case report: https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260 https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260 Relevant section: > Based on the timeline of this case, it appears that the patient either consulted ChatGPT 3.5 or 4.0 when considering how he might remove chloride from this diet. Unfortunately, we do not have access to his ChatGPT conversation log and we will never be able to know with certainty what exactly the output he received was, since individual responses are unique and build from previous inputs. > However, when we asked ChatGPT 3.5 what chloride can be replaced with, we also produced a response that included bromide. Though the reply stated that context matters, it did not provide a specific health warning, nor did it inquire about why we wanted to know, as we presume a medical professional would do.
pahkah 1y ago
This seems like a case of tunnel vision and confirmation bias, the nasty combo that sycophantic LLMs make easy to fall prey to. Someone gets an idea, asks about it, and the LLM doesn’t ask about the context or say that doesn’t make sense, it just plays along, “confirming” that that the idea was correct. I’ve caught myself with this a few times when I sort of suggest a technical solution that, in hindsight, was the wrong way to approach a problem. The LLM will try to find a way to make that work without taking a step back and suggesting that I didn’t understand the problem I was looking at.
bitwize 1y ago
In his poem "The Raven", Edgar Allan Poe's narrator knows, at least subconsciously, that the bird will respond with "nevermore" to whatever is asked of it. So he subconsciously formulates his queries to it in such a way that the answers will deepen his madness and sorrow. People are starting to use LLMs in a similar fashion -- to confirm and thus magnify whatever wrong little notion they have in their heads until it becomes an all-consuming life mission to save the world. And the LLMs will happily oblige because they aren't really thinking about what they're saying, just choosing tokens on a mathematical "hey, this sounds nice" criterion. I've seen this happen with my sister, who is starting to take seriously the idea that ChatGPT was actually created 70 years ago at DARPA based on technology recovered from the Roswell crash, based on her conversations with it. I can't really blame the LLMs entirely for this, as like the raven they're unwittingly being used to justify whatever little bit of madness people have in them. But we all have a little bit of madness in us, so this motivates me further to avoid LLMs entirely, except maybe for messing around.
willguest 1y ago
I think an award ceremony would be the best way to draw attention to the outrageous implications of blindly following artificial "intelligence" to wherever it may lead you. something like the Darwin awards, but dedicated to clanker wankers, a term I am coining for those people who are so self-absorbed that they feel a dispropotionate sense of validation from a machine that is programmed to say "you're absolutely right" at every available juncture. That's not to say this isn't rooted in mental illness, or perhaps a socially-motivated state of mind that causes a total absence of critical thinking, but some kind of noise needs to be made and I think public (ahem) recognition would be a good way to go. A catchy name is essential - any suggestions?
- polonbike 1y ago
  A-Idiocraty
- water-data-dude 1y ago
  The Navel Klein Bottle Award
puppycodes 1y ago
People have been giving bad health advice for all of human history including now. Talking about AI like its sentient and a monolith is the problem. It's like saying computers give bad health advice because the internet.

59 points by 01-_- 1y ago

>for the past three months, he had been replacing regular table salt with sodium bromide. His motivation was nutritional—he wanted to eliminate chloride from his diet, based on what he believed were harmful effects of sodium chloride. Ok so the article is blaming chatgpt but this is ridiculous. Where do you buy this bromide? It's not like it's in the spices aisle. The dude had to go buy a hot tub cleaner like Spa Choice Bromine Booster Sodium Bromide and then sprinkle that on his food. I dont care what chatgpt said... that dude is the problem.

bluefirebrand 1y ago
> I dont care what chatgpt said... that dude is the problem. This reminds me of people who fall for phone scams or whatever. Some number of the general population is susceptible to being scammed, and they wind up giving away their life's savings or whatever to someone claiming to be their grandkid There's always an asshole saying "well that's their own fault if they fall for that stuff" as if they chose it on purpose instead of being manipulated into it by other people taking advantage of them See it a lot on this site, too. How clever are the founders who exploit their workers and screw them out of their shares, how stupid are the workers who fell for it
- deleted 1y ago
  [deleted]
- crazygringo 1y ago
  > See it a lot on this site, too. How clever are the founders who exploit their workers and screw them out of their shares, how stupid are the workers who fell for it I have literally never seen that expressed on HN. In every case where workers are screwed out of shares, the sympathy among commenters seems to be 100% with the employees. HN is pretty anti-corporate overall, if you haven't noticed. Yes it's pro-startup but even more pro-employee.
  - bluefirebrand 1y ago
    > HN is pretty anti-corporate overall, if you haven't noticed. My observation is that any given thread can go either way, and it sometimes feels like a coin toss which side of HN will be most represented in any given thread Yes, I have seen quite a lot of anti-corporate posts, but I also see quite a few anti-employee posts. This is likely my own negative bias but I think many users here are generally pro-Capital which aligns them with corporate interests even if they are some degree of anti-corporate anyways Probably I just fixate too much on the posts I have a negative reaction to
    - crazygringo 1y ago
      What kinds of anti-employee posts have you seen? I'm genuinely curious, because I can't think of any. But I'm wondering if maybe I'm mentally categorizing posts differently from you?
- rowanG077 1y ago
  Both can be true at the same time. You can be an idiot to fall far a scam while the scammer is a dickhead criminal.
  - bluefirebrand 1y ago
    Scammers only work because they know some percentage of the population is going to fall for it Can't we have some empathy for people just trying to do their best in a world where so many people are trying to take advantage of them? Their victims are often the vulnerable ones in our society too. The elderly, the infirm, the mentally ill. It's not just "stupid people fall for scams" it takes one lapse of judgement over a lifetime of being targeted. Come on
    - rowanG077 1y ago
      Of course people need to be protected. That's why it's literally illegal to scam people almost everywhere. That doesn't men that people should not ALSO attempt to protect themselves from scammers. It's quite dirty to bring up the elderly, infirm and mentally ill. Because of course they cannot help themselves. Those groups are not what this is about, and you damn well know it. This is about normal functioning adults walking into scams with their eyes open. And yes that group has a responsibility to keep up with the scams that are commonplace. It's ridiculous to encourage people to go through life with their blinders on because "the world should just be a fair place". Yeah it should be, but tough luck, reality is different.
      - bluefirebrand 1y ago
        > Those groups are not what this is about, and you damn well know it. This is about normal functioning adults walking into scams with their eyes open Normal functioning adults will also benefit if we take steps to protect the infirm and dysfunctional That's why it isn't meant to be "dirty" to bring up the vulnerable in society. If we take sufficient steps to protect them, we all benefit
        1y ago
jazzyjackson 1y ago
Agreed, but, it is in the spices aisle if your spice aisle is amazon.com
- throwaway173738 1y ago
  You haven’t lived if you haven’t tried my Bromine Chicken. I make it every Christmas.
beardyw 1y ago
> Ok so the article is blaming chatgpt but this is ridiculous. People are born with certain attributes. Some are born tall, some left handed and some gullible. None of those is a reason to criticise them.
dinfinity 1y ago
We could also point the finger towards the popular consensus that "salt is bad for you". This guy just took it to the next level.

infecto 1y ago

Have a large part of the population always been susceptible to insane conspiracies and psychosis or is this recent phenomenon? The feels less of a ChatGPT problem and something more is at play.

morkalork 1y ago
Yes. I see this is as the digital equivalent of the people who convince themselves to believe that colloidal silver will heal their ailments and turn themselves blue
jalk 1y ago
I.e. Bleach against covid edit: A quick google search shows, there is no evidence of anybody actually in-gesting/jecting bleach to fight COVID
- philipallstar 1y ago
  [flagged]
  - mort96 1y ago
    Not sure what exactly you use "patriarchy" to mean here, but the idea that positions of power and prestige in society are occupied more by men than by women is kind of just a statistical fact. Look at the gender ratios of CEOs in the Fortune 500, or the gender ratios of US presidents, or any number of other positions of power
    - philipallstar 1y ago
      This is a motte and bailey fallacy. The reason this difference has an ominous-sounding name, or a name at all, is because there's an implicit "this is a global conspiracy against women" attached.
  - jmkd 1y ago
    What many misunderstand about the patriarchy is in thinking its an agreed, secret, organised system of ensuring male dominance. That perspective is so easy to refute, this refutation gets equated with the non-existence of the patriarchy itself. But the patriarchy is simply an unagreed, mostly transparent and organic coincidence of male dominance. There is no mysterious cabal of men saying let's pay men more than women, but men are paid more than women anyway. Likewise there has never been a female US president, or chess world champion, or F1 champion, which does not actually point to talent but to inequality. You might say the patriarchy is not a diagnosis, but a symptom. The facts of male advantage in society are irrefutable, and if you consider that they permeate every sphere from public to private, social to corporate, across age and culture for many thousands of years, it doesn't hold water to claim this symptom doesn't exist. It's just not a conspiracy theory, that's all.

hereme888 1y ago

The man followed insane health advice given by GPT 3.5. we're at v5. Very outdated report.

deleted 1y ago
[deleted]
headinsand 1y ago
[flagged]

moduspol 1y ago

I continue to be surprised that LLM providers haven't been legally cudgeled into neutering the models from ever giving anything that can be construed as medical advice. I'm glad--I think LLMs are looking quite promising for medical use cases. I'm just genuinely surprised there's not been some big lawsuit yet over it providing some advice that leads to some negative outcome (whether due to hallucinations, the user leaving out key context, or something else).

qwertylicious 1y ago
This is the story of the modern tech industry at large: a major new technology is released, harms are caused, but because of industry norms and a favourable legal environment, companies aren't held liable for those harms. It's pretty amazing, really. Build a washing machine that burns houses down and the consequences are myriad and severe. But build a machine that allows countless people's private information to be leaked to bad actors and it's a year of credit monitoring and a mea culpa. Build a different machine that literally tells people to poison themselves and, not only are there no consequences, you find folks celebrating that the rules aren't getting in the way. Go figure.
- usrnm 1y ago
  How many people do you think the early steam engines killed? Or airplanes
  - qwertylicious 1y ago
    Or sweatshops or radium infused tinctures. We've moved on from the 1800s. Why are you using that as your baseline of expectation?
    - api 1y ago
      There's a very common belief that things like regulations and especially liability simply halts all innovation. You can see some evidence for this point of view from aerospace with its famous "if it hasn't already flown, it can't fly" mentality. It's why we are still using leaded gasoline in small planes, though this is finally being phased out... but it took an unreasonably long time due to certification requirements and bureaucracy. If airplanes weren't so heavily regulated we'd have seen leaded gasoline vanish there around the same time it did in cars, but you also might have had a few crashes due to engine failures as the bugs were worked out with changes and retrofits. I'm a little on the fence here. I don't want a world where we basically conduct human sacrifice for progress, but I also don't want a world that is frozen in time. We really need to learn how to have responsible, careful progress, but still actually do things. Right now I think we are bad at this. Edit: I think it's related to some extent to the fact that nuanced positions are hard in politics. In popular political discourse positions become more and more simple, flat, and extreme. There's a kind of context collapse that happens when you try to scale human organizations, what I want to call "nuance collapse," that makes it very hard to do anything but all A or all B. For innovation it's "full speed ahead" vs "stop everything."

Workaccount2 1y ago

I expect a barage of these headline grabbing long tail stories being pushed out of psychology circles as more and more people find ChatGPT more helpful than their therapist (which is already becoming very popular). We need to totally ban LLMs from doing therapy like conversations, so that a pinch of totally unhinged people don't do crazy stuff. And of course everyone needs to pay a human for therapy to stop this.

davidmurdoch 1y ago
Is your last paragraph sarcasm?
- Workaccount2 1y ago
  Firmly

deleted 1y ago

[deleted]

permo-w 1y ago

>his exact interaction with ChatGPT remains unverified there is no informational point to this article if the entire crux is 'the patient wanted to eat less "chloride" and claims ChatGPT told him about Sodium Bromide". based on this article, the interaction could have been as minimal as the guy asking for the existence of an alternative salt to sodium chloride, unqualified information he equally could have found on a chemistry website or wikipedia

throwaway173738 1y ago
Yeah you found the paragraph where they highlight that there don’t know what interaction with ChatGPT gave him that information. The reason they’re sharing the anecdote is because there might be a new trend developing in medicine where people go to the ED after taking advice from an LLM that leads to injury and maybe screening questions should include asking about that.
- permo-w 1y ago
  and yet this doesn't change the fact that they wrote an entire medical article the crux of which is little more than hearsay. "did you get advice from an LLM?" is far less relevant and all-catching a question here than "have you made any dietary changes recently?" and yet the article isn't about that, because odd dietary changes aren't the attention-grabbing topic right now. I imagine you could find thousands of similar stories where the culprit was google or facebook or youtube instead of an LM, and yet nothing needs to be changed for them because they too can be covered with a question akin to "have you made any dietary changes recently?"
  - throwaway173738 1y ago
    If there was a guy out there driving around selling bromide tablets to people as a substitute for dangerous chloride in your biochemistry I think asking if you’ve bought anything from the back of a wagon is a reasonable response. Doctors as a group often try to solve health problems by looking for societal trends. It’s how a lot of diseases get spotted. They’re not saying that using an LLM is the dangerous thing, they’re saying there might be some correlation between soliciting advice from the machine and unusual conditions and it merits further study, so please ask your patients.

amiga386 1y ago

Oh no, the man used the hallucination engine, which told the man, in a confident tone, a load of old twaddle. The hallucination engine doesn't know anything about what it told the man, because it neither knows nor thinks things. It's a data model and an algorithm. The humans touting it and bigging it up, so they'll get money, are the problem.

jdonaldson 1y ago
Humans make mistakes too. Case in point, the hallucination engine didn't tell the person to ingest bromide. It only mentioned that it had chemical similarities to salt. The human mistakenly adopted a bit of information that furthered his narrative. The humans touting and bigging it up are still the problem.
- thunderfork 1y ago
  We don't have the log from this case, so we don't know what chatgippity said, whether it was "chemical similarities" or "you should consume bromium... now!"
- wzdd 1y ago
  Could you provide a source for your statements? The article says that they don’t have access to the chat logs, and the quotes from the patient don’t suggest that chatgpt did not tell him to ingest bromide.

DiabloD3 1y ago

As a reminder to the wider HN: LLMs are only statistical models. They cannot reason, they cannot think, they can only statistically (and non-factually) reproduce what they were trained on. It is not an AI. This person, sadly, and unfortunately, gaslit themselves using the LLM. They need professional help. This is not a judgement. The Psypost article is a warning to professionals more than it is anything else: patients _do_ gaslight themselves into absurd situations, and LLMs just help accelerate that process, but the patient had to be willing to do it and was looking for an enabler and found it in an LLM. Although I do believe LLMs should not be used for "chat" models, and only explicitly, on-rails, text completion and generation tasks (in the functional lorem ipsum sense), this does not actually seem to be the fault of the LLM directly. I think providers should be forced to warn users that LLMs cannot factually reproduce anything, but I think this person would have still weaponized LLMs against themselves, and this would have been the same outcome.

simonw 1y ago

Here's the clinical case report: https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260 https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260 Relevant section: > Based on the timeline of this case, it appears that the patient either consulted ChatGPT 3.5 or 4.0 when considering how he might remove chloride from this diet. Unfortunately, we do not have access to his ChatGPT conversation log and we will never be able to know with certainty what exactly the output he received was, since individual responses are unique and build from previous inputs. > However, when we asked ChatGPT 3.5 what chloride can be replaced with, we also produced a response that included bromide. Though the reply stated that context matters, it did not provide a specific health warning, nor did it inquire about why we wanted to know, as we presume a medical professional would do.

pahkah 1y ago

This seems like a case of tunnel vision and confirmation bias, the nasty combo that sycophantic LLMs make easy to fall prey to. Someone gets an idea, asks about it, and the LLM doesn’t ask about the context or say that doesn’t make sense, it just plays along, “confirming” that that the idea was correct. I’ve caught myself with this a few times when I sort of suggest a technical solution that, in hindsight, was the wrong way to approach a problem. The LLM will try to find a way to make that work without taking a step back and suggesting that I didn’t understand the problem I was looking at.

bitwize 1y ago

In his poem "The Raven", Edgar Allan Poe's narrator knows, at least subconsciously, that the bird will respond with "nevermore" to whatever is asked of it. So he subconsciously formulates his queries to it in such a way that the answers will deepen his madness and sorrow. People are starting to use LLMs in a similar fashion -- to confirm and thus magnify whatever wrong little notion they have in their heads until it becomes an all-consuming life mission to save the world. And the LLMs will happily oblige because they aren't really thinking about what they're saying, just choosing tokens on a mathematical "hey, this sounds nice" criterion. I've seen this happen with my sister, who is starting to take seriously the idea that ChatGPT was actually created 70 years ago at DARPA based on technology recovered from the Roswell crash, based on her conversations with it. I can't really blame the LLMs entirely for this, as like the raven they're unwittingly being used to justify whatever little bit of madness people have in them. But we all have a little bit of madness in us, so this motivates me further to avoid LLMs entirely, except maybe for messing around.

willguest 1y ago

I think an award ceremony would be the best way to draw attention to the outrageous implications of blindly following artificial "intelligence" to wherever it may lead you. something like the Darwin awards, but dedicated to clanker wankers, a term I am coining for those people who are so self-absorbed that they feel a dispropotionate sense of validation from a machine that is programmed to say "you're absolutely right" at every available juncture. That's not to say this isn't rooted in mental illness, or perhaps a socially-motivated state of mind that causes a total absence of critical thinking, but some kind of noise needs to be made and I think public (ahem) recognition would be a good way to go. A catchy name is essential - any suggestions?

polonbike 1y ago
A-Idiocraty
water-data-dude 1y ago
The Navel Klein Bottle Award

puppycodes 1y ago

People have been giving bad health advice for all of human history including now. Talking about AI like its sentient and a monolith is the problem. It's like saying computers give bad health advice because the internet.