5 ms·

I don’t think they will improve, there is too much incentive to poison the datasets going forward. A lot of the models up to this point have been benefitted -

by lazide 4mo ago

I don’t think they will improve, there is too much incentive to poison the datasets going forward. A lot of the models up to this point have been benefitted - like Google did - from essentially ‘pre SEO’ internet. Now the same tools are being used to generate nigh infinite good sounding bullshit, which poisons the dataset in all sorts of hard to detect ways. To add insult to injury, the human experts are also not as. Naive, and have many incentives to poison their own input in subtle ways too.

rvnx 4mo ago
Human doctors use LLMs to diagnose too OpenEvidence claims "More than 40% of U.S. physicians use it daily, and it handled around 20 million clinical consultations per month. Over 100 million Americans were treated by a doctor using it in 2025." https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doctors-doubles-valuation-to-12-billion.html https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doc...
- something98 4mo ago
  This is a very misleading statement; most of those physicians are using LLMs to transcribe notes from visits and/or for billing purposes (e.g., proper billing codes).
  - brokencode 4mo ago
    OpenEvidence is specifically meant to help clinicians make evidence-based decisions in the diagnosis and treatment of patients, not note transcription.
    - sxg 4mo ago
      It does both: https://www.openevidence.com/user-guide/visits-overview https://www.openevidence.com/user-guide/visits-overview
  - kjellsbells 4mo ago
    The problems isnt LLMs per se, it is the shift to trusting the output of the machine coupled with a decline in verifying that the output is reasonable. It's basically what your teachers warned you about with wikipedia in eight grade except applied to all areas of life, including medicine. Dictation is already high-stakes and LLMs do not automatically reduce that risk. Here is an example. My provider sent me this note. I'm quoting verbatim here from my MyChart record: "Your liver enzymes are high, I would like to order acetaminophen containing medication like Tylenol, I would like to order liver ultrasound I placed ultrasound order in the system, make an appointment for radiology, I would like you to get hepatitis panel lab work done, obtain blood work order, please schedule a well visit to get it done" When I queried it, this is what I got back. It was a dictation error. You could almost hear the panic in the message: "Sorry for wrong message earlier, I was dictated message- so could not realize that it was written to take Tylenol type of medicines- I DO NOT RECOMMEND ACETAMINOPHEN CONTAINING MEDICINE - LIKE TYLENOL AND ALCOHOL DUE TO ELEVATED LIVER ENZYMES." Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.
    - ethbr1 4mo ago
      > Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine. 100%. Also, management. I wish someone would go ahead and coin an AI version of Amdahl's law that states the work speedup from AI is dependent on amount of unverified AI output used. Iow, if you 1:1 verified everything, there would be no time savings. Ergo, you get management saying (1) we demand time savings due to AI & (2) we demand you fully check anything you use AI for. End result? People skip (2) to hit (1). Then management burns anyone at the stake whenever inevitable mistakes happen.
      - lazyasciiart 4mo ago
        But that’s trivially false. There is an entire category of work where it is hard to come up with an answer and easy to verify the answer, which means that if you verified everything there would still be a large time savings.
  - girvo 4mo ago
    Which is itself a problem as (in my partners evaluations as an optometrist), LLMs used for clinical notes has a bad habit of dropping clinically important information, and the biggest providers don’t give you a copy of the raw transcript or a recording Which means she ends up spending just as much time as if she’d done it herself as it needs to be verified for accuracy every time…
- sarchertech 4mo ago
  Ignoring the fact that this number comes from a company press release, it doesn’t say anything about the number of doctors using it to diagnose, just that they use it. If a physician uses Google to search for a dosage chart for some drug they rarely prescribe, you wouldn’t say they are using Google to diagnose the patient. You wouldn’t say that either if they used Google to search for the most recent studies on a topic.
- sambellll 4mo ago
  To me this is like a good software engineer using AI. The fact that they use it doesn't make what the result is any worse or less trustworthy - arguably it makes it better. It only becomes a problem if they offload all of the thinking to AI.
brokencode 4mo ago
I seriously doubt that data set poisoning will be a real limiter in model performance. For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models? For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc. Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.
- microgpt 4mo ago
  Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
  - thfuran 4mo ago
    >Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
    - microgpt 4mo ago
      yeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't. and browsers forcibly auto-update
- Analemma_ 4mo ago
  I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.

by lazide 4mo ago

rvnx 4mo ago

Human doctors use LLMs to diagnose too OpenEvidence claims "More than 40% of U.S. physicians use it daily, and it handled around 20 million clinical consultations per month. Over 100 million Americans were treated by a doctor using it in 2025." https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doctors-doubles-valuation-to-12-billion.html https://www.cnbc.com/2026/01/21/openevidence-chatgpt-for-doc...

something98 4mo ago
This is a very misleading statement; most of those physicians are using LLMs to transcribe notes from visits and/or for billing purposes (e.g., proper billing codes).
- brokencode 4mo ago
  OpenEvidence is specifically meant to help clinicians make evidence-based decisions in the diagnosis and treatment of patients, not note transcription.
  - sxg 4mo ago
    It does both: https://www.openevidence.com/user-guide/visits-overview https://www.openevidence.com/user-guide/visits-overview
- kjellsbells 4mo ago
  The problems isnt LLMs per se, it is the shift to trusting the output of the machine coupled with a decline in verifying that the output is reasonable. It's basically what your teachers warned you about with wikipedia in eight grade except applied to all areas of life, including medicine. Dictation is already high-stakes and LLMs do not automatically reduce that risk. Here is an example. My provider sent me this note. I'm quoting verbatim here from my MyChart record: "Your liver enzymes are high, I would like to order acetaminophen containing medication like Tylenol, I would like to order liver ultrasound I placed ultrasound order in the system, make an appointment for radiology, I would like you to get hepatitis panel lab work done, obtain blood work order, please schedule a well visit to get it done" When I queried it, this is what I got back. It was a dictation error. You could almost hear the panic in the message: "Sorry for wrong message earlier, I was dictated message- so could not realize that it was written to take Tylenol type of medicines- I DO NOT RECOMMEND ACETAMINOPHEN CONTAINING MEDICINE - LIKE TYLENOL AND ALCOHOL DUE TO ELEVATED LIVER ENZYMES." Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine.
  - ethbr1 4mo ago
    > Again the problem is not dictation, or LLMs. The problem is humans ignoring their responsibility to check the output of a machine. 100%. Also, management. I wish someone would go ahead and coin an AI version of Amdahl's law that states the work speedup from AI is dependent on amount of unverified AI output used. Iow, if you 1:1 verified everything, there would be no time savings. Ergo, you get management saying (1) we demand time savings due to AI & (2) we demand you fully check anything you use AI for. End result? People skip (2) to hit (1). Then management burns anyone at the stake whenever inevitable mistakes happen.
    - lazyasciiart 4mo ago
      But that’s trivially false. There is an entire category of work where it is hard to come up with an answer and easy to verify the answer, which means that if you verified everything there would still be a large time savings.
- girvo 4mo ago
  Which is itself a problem as (in my partners evaluations as an optometrist), LLMs used for clinical notes has a bad habit of dropping clinically important information, and the biggest providers don’t give you a copy of the raw transcript or a recording Which means she ends up spending just as much time as if she’d done it herself as it needs to be verified for accuracy every time…
sarchertech 4mo ago
Ignoring the fact that this number comes from a company press release, it doesn’t say anything about the number of doctors using it to diagnose, just that they use it. If a physician uses Google to search for a dosage chart for some drug they rarely prescribe, you wouldn’t say they are using Google to diagnose the patient. You wouldn’t say that either if they used Google to search for the most recent studies on a topic.
sambellll 4mo ago
To me this is like a good software engineer using AI. The fact that they use it doesn't make what the result is any worse or less trustworthy - arguably it makes it better. It only becomes a problem if they offload all of the thinking to AI.

brokencode 4mo ago

I seriously doubt that data set poisoning will be a real limiter in model performance. For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models? For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc. Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.

microgpt 4mo ago
Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit.
- thfuran 4mo ago
  >Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.
  - microgpt 4mo ago
    yeah but unless everyone is checking the version, if it's just a handful of websites checking it, they don't. and browsers forcibly auto-update
Analemma_ 4mo ago
I think you underestimate just how much money is being poured into LLM SEO at the moment. It's real quiet because they don't want to draw attention and countermeasures from the frontier labs, but this is getting huge investment, and they will have a monomaniac focus on juicing product results whereas the attention of the labs necessarily has to be spread out.