7 ms·
This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because t
by appplication 3mo ago
This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief.
It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.
Don’t get me wrong, I think we all agree capabilities will eventually improve (and farther-future capabilities could reasonably surpass experts), but really is unclear if the current transformer architectures with their probabilistic/hallucinatory outputs will plateau before they surpass current experts abilities in all promised fields.
- lazide 3mo agoI don’t think they will improve, there is too much incentive to poison the datasets going forward. A lot of the models up to this point have been benefitted - like Google did - from essentially ‘pre SEO’ internet. Now the same tools are being used to generate nigh infinite good sounding bullshit, which poisons the dataset in all sorts of hard to detect ways. To add insult to injury, the human experts are also not as. Naive, and have many incentives to poison their own input in subtle ways too.
- rvnx 3mo agoHuman doctors use LLMs to diagnose too OpenEvidence claims "More than 40% of U.S. physicians use it daily, and it handled around 20 million clinical consultations per month. Over 100 million Americans were treated by a doctor using it in 2025." https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doctors-doubles-valuation-to-12-billion.html https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doc...
- something98 3mo agoThis is a very misleading statement; most of those physicians are using LLMs to transcribe notes from visits and/or for billing purposes (e.g., proper billing codes).
- brokencode 3mo agoOpenEvidence is specifically meant to help clinicians make evidence-based decisions in the diagnosis and treatment of patients, not note transcription.
- sxg 3mo agoIt does both: https://www.openevidence.com/user-guide/visits-overview https://www.openevidence.com/user-guide/visits-overview
- kjellsbells 3mo agoThe problems isnt LLMs per se, it is the shift to trusting the output of the machine coupled with a decline in verifying that the output is reasonable. It's basically what your teachers warned you about with wikipedia in eight grade except applied to all areas of life, including medicine. Dictation is already high-stakes and LLMs do not automatically reduce that risk. Here is an example. My provider sent me this note. I'm quoting verbatim here from my MyChart record: "Your liver enzymes are high, I would like to order acetaminophen containing medication like Tylenol, I would like to order liver ultrasound I placed ultrasound order in the system, make an appointment for radiology, I would like you to get hepatitis panel lab work done, obtain blood work order, please schedule a well visit to get it done" When I queried it, this is what I got back. It was a dictation error. You could almost hear the panic in the message: "Sorry for wrong message earlier, I was dictated message- so could not realize that it was written to take Tylenol type of medicines- I DO NOT RECOMMEND ACETAMINOPHEN CONTAINING MEDICINE - LIKE TYLENOL AND ALCOHOL DUE TO ELEVATED LIVER ENZYMES." Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.
- ethbr1 3mo ago> Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine. 100%. Also, management. I wish someone would go ahead and coin an AI version of Amdahl's law that states the work speedup from AI is dependent on amount of unverified AI output used. Iow, if you 1:1 verified everything, there would be no time savings. Ergo, you get management saying (1) we demand time savings due to AI & (2) we demand you fully check anything you use AI for. End result? People skip (2) to hit (1). Then management burns anyone at the stake whenever inevitable mistakes happen.
- sarchertech 3mo agoIgnoring the fact that this number comes from a company press release, it doesn’t say anything about the number of doctors using it to diagnose, just that they use it. If a physician uses Google to search for a dosage chart for some drug they rarely prescribe, you wouldn’t say they are using Google to diagnose the patient. You wouldn’t say that either if they used Google to search for the most recent studies on a topic.
- sambellll 3mo agoTo me this is like a good software engineer using AI. The fact that they use it doesn't make what the result is any worse or less trustworthy - arguably it makes it better. It only becomes a problem if they offload all of the thinking to AI.
- brokencode 3mo agoI seriously doubt that data set poisoning will be a real limiter in model performance. For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models? For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc. Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.
- microgpt 3mo agoPretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
- thfuran 3mo ago>Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
- microgpt 3mo agoyeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't. and browsers forcibly auto-update
- Analemma_ 3mo agoI think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.
- sublinear 3mo agoHuman expertise is also improving all the time and not limited to just connecting dots. When AI seems to surpass a particular human, it's just because the human lacks broader knowledge and fails to investigate further. An expert already knows they don't know everything. That was never the point. Critical thinking cannot be delegated to AI any more than it can be delegated to a book. There is nothing new going on here.
- cheschire 3mo agoI was a very early adopter in my circles with AI and I shared it with many people. Strangely, I seem to be the most skeptical about AI in my circles as well, but because I was the gateway for a many folks, they want to come back and share their experiences with me. And it's so much like listening to someone in a church congregation sharing their experiences with god. Clear and obvious gaps are hand-waved away exactly how you're describing.
- TomasBM 3mo agoWhy is it psychosis and not lower standards? While I can understand being skeptical of non-experts' claims that such answers are enough, I don't understand why you call it "psychosis" and not simply naivety or lack of expertise. At the same time, the new so-called "models" haven't been pure transformer-based LLMs, but entire systems with tools (with access to the Internet), data storage, and the options to trigger additional instances for different tasks.
- janmatejka 3mo agoBecause some people develop actual psychosis. They go down some rabbit hole with an LLM until the LLM makes them believe they invented new kind of physics that makes them go harassing experts who obviously try to ignore them because its all nonsense.
- ruszki 3mo agoFor me, what others said and literally showed with Claude Code, et al, and what I’ve been experiencing with it, clearly signal way lower standards. But this was true even before LLMs.
- shimman 3mo agoReminds me of that clip of Travis Kalanick, sexual deviant and harasser of women, talking about "discovering new physics."
- natebc 3mo agoThe Uber guy? Yeah that was a painful watch.
- perching_aix 3mo agoGraciously diagnosed for them by random unqualified people on the internet with an agenda, frequently before even any relevant interaction: "Oh you like LLMs? You must in AI psychosis!" Let's not pretend it is anything more than the run of the mill wet fart of a culture war label. It's quite literally the "TDS" of the anti-AI crowd.
- operatingthetan 3mo ago>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem. The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.
- singpolyma3 3mo agoThat presumes that we have a definition of "thinking" or that we know that anything is "thinking" when in fact neither is true. The problem is real but I don't think positing a philosophical root is helpful
- operatingthetan 3mo agoThe claim that we are assigning human-like agency to a machine with none is simple and factual.
- ForceBru 3mo agoWhat's "thinking"? What's "agency"? What's "human-like agency"? If "agency" is making decisions and performing corresponding actions in the real world, then LLMs most definitely LOOK LIKE they're making decisions (what's the next token? which tool to use? what's to say, in general? what idea to convey?) and performing actions (tool use). Can we tell whether they are ACTUALLY making decisions? Well, are the people around me "actually" making decisions? Or are they simply pushed around by circumstances and external forces? Am I actually making decisions? Did I like DECIDE to write this comment? Maybe? I have no clue...
- operatingthetan 3mo agoI think you're mildly obfuscating the issues at hand by diving too deeply into philosophical questions. It's quite simple, the agency that the LLM appears to have is actually your own. Without a prompt an LLM does nothing. It has no thoughts between prompts about you or your problems.
- deleted 3mo ago[deleted]
- perching_aix 3mo ago> There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks Do you think it is any more possible to have a proper discussion with someone who preemptively paints the other person as mentally ill? Or someone who preemptively victimizes themselves? Cause I don't think these are the hallmarks of an honest discussion. See also the entire past decade of political discourse. Like, consider this: > It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities. A trivial counter to this is that you can just be an expert at something (e.g. your own work), use the damn thing yourself (professionally), and evaluate the outcomes for yourself. Then maybe remark "LLM good". Now you come and remark "LLM bad", and point at random "evidence", either of outright other workloads, or even the one at hand: you're asking someone to reject the reality they've already experienced, entirely based on the assumption that they're "merely religious" or "in psychosis". You tell me if that's any more epistemically rigorous and sensible than their story.
- hectdev 3mo agoI _believe_ the term "AI Psychosis" is a "thought-terminating cliché" that readily puts you in a position to disarm any criticism to your point of view, which, if you're aware of it or not, it's a belief in and of itself. I'm more willing to bet your can't have discussions because you're trying to have debates. But on your actual point, I don't think AI needs to "surpass current experts abilities in all promised fields" as a marker of its ability. The immediate gains has already shown some remarkable promise and more LLMs should have had safeguards around mental health up front. If I were to put it on a scale, I would say it is net positive long term with a strong negative spike up front which was somewhat preventable. But who knows, maybe is just have "AI Psychosis" and you can easily dismiss me.