8 ms·

Google removes AI health summaries

226 points by barishnamazov 9mo ago

dreadsword 9mo ago
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them... https://deadstack.net/cluster/google-removes-ai-overviews-for-medical-queries https://deadstack.net/cluster/google-removes-ai-overviews-fo...
- bandrami 9mo ago
  Why does healthcare "need disruption"?
  - miltonlost 9mo ago
    The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
    - zdragnar 9mo ago
      The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
      - kunai 9mo ago
        Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
    - eru 9mo ago
      Profit is a small part of overall revenue.
  - wswin 9mo ago
    It's inefficient and not living to its potential
    - bandrami 9mo ago
      And "disruption" (a pretty ill-defined term) is the solution to that?
      - __loam 9mo ago
        The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
        gradus_ad 9mo ago
        There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here. And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
        __loam 9mo ago
        We'd rather have medical bankruptcy than a functional system.
        ikr678 9mo ago
        Maybe for R&D but your outsized costs are also due to your liability environment and direct incentives against preventative healthcare.
    - plagiarist 9mo ago
      The inefficiency is the buying of yachts for billionaires.
  - dyauspitr 9mo ago
    Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
    - gmueckl 9mo ago
      But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
      - eru 9mo ago
        Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance? Profit isn't even a big part of the overall revenue. > Mandate at least decent minimal coverage standards I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices. > and large insurance pools that must span age groups and risk groups. Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
        FireBeyond 9mo ago
        > Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you). Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
  - harpratap 9mo ago
    Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
    - eru 9mo ago
      I sympthatise with the argument. We should test it against real world data. Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
      - eru 9mo ago
        I just looked it up, and apparently health care costs for pets has gone up in price even faster than for humans. Pets typically don't have medical insurance, and any insurance that does exist there has a radically different regulatory regime than for humans. Since 1980 for the US: CPI has gone up by 3.16% on average per year (x4.17 in total). Human healthcare costs by 4.9% per year (x8.96 in total). And pet healthcare costs by 6.49% (or x17.87 in total).
        manuelmoreale 9mo ago
        I suspect this has something to do with it: https://www.theatlantic.com/ideas/archive/2024/04/vet-private-equity-industry/678180/ https://www.theatlantic.com/ideas/archive/2024/04/vet-privat...
        eru 9mo ago
        That could at most explain the last few years of rising vet prices.
        bandrami 9mo ago
  - sergiotapia 9mo ago
    because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
    - sarchertech 9mo ago
      Physician compensation is around 9% of healthcare spending. The number of non-physician providers (NPs, PAs and specialists like physical therapists and podiatrists) has also exploded over the last 20 years. We have far more overall providers per capita than we did 20 years ago. Lack of providers isn’t what’s driving up costs.
      - derbOac 9mo ago
        I don't think physician compensation per se is a good metric for capturing the effect of lack of providers, because some of the increased costs are due to the bottlenecks in the services per se, in terms of procedure costs and types of procedures offered. I also don't think the number of providers per se under the current regime, without deregulation or reregulation of practice boundaries, is representative of what would happen if there were changes in those boundaries. Adding more optometrists 5 years ago isn't the same as changing what they're allowed to do. It also doesn't address what cost increases would have been without an increase in the number of providers. 9% might also seem pretty big to me if it's out of all spending and doesn't include other provider compensation? What if overall healthcare costs went down, but physician compensation stayed the same? Would that then be a problem because it was an increased proportion of the total costs — fat left to be trimmed, so to speak? There are many problems that don't have anything to do with providers per se, but I also don't think you can glean much by extrapolating to more of the same, especially compensation per se.
        thepotatodude 9mo ago
        9% is nothing when you consider that these people provide the value. With no physicians, there is no healthcare.
        9mo ago
  - seanmcdirmid 9mo ago
    Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
    - scarab92 9mo ago
      Society pretends that human doctors are better than they really are, and AI is worse than it really is. It's the self-driving cars debate all over again.
      - thepotatodude 9mo ago
        It's really just not there yet. I've been in medical school for >3 years now and have been using the latest models with good prompting. They have gotten much better, but I still see misses that my classmates would easily catch. This is not acceptable in healthcare. It's certainly not getting 100% on all my assignments, which are a step below the complexity of real-world clinical practice. Before medical school, I was not so sure of the quality of your average doc. Now having spent a year in clinical practice across various settings, I am extremely reassured. I can say with certainty that a US trained doctor is miles ahead of AI right now. The system sucks really bad though and forces physicians to churn patients, giving the impression that physicians don't pay attention/don't care/etc.
    - bandrami 9mo ago
      I mean, if we're talking Christensenian disruption, that happens in neglected markets rather than currently dysfunctional ones. There's no shortage of actors wronging money out of health care so there's not a disruptable space per se.
      - omnicognate 9mo ago
        > wronging Not sure if this is a typo (of wringing) or a pun, but it's apt either way.
xnx 9mo ago
Ars rips of this original reporting, but makes it worse by leaving out the word "some" from the title. ‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk: https://www.theguardian.com/technology/2026/jan/11/google-ai-overviews-health-guardian-investigation https://www.theguardian.com/technology/2026/jan/11/google-ai...
- miltonlost 9mo ago
  Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk" Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"
TwoNineA 9mo ago
[flagged]
- moss_dog 9mo ago
  What's that?
  - BloodyIron 9mo ago
    lol you must be fresh to the internet
jnamaya 9mo ago
chatGPT told me, I am the healthiest guy in the world, and I believe it
- deleted 9mo ago
  [deleted]
- xxs 9mo ago
  If it's any help women appreciate confidence; Yet, the article is not about chatgpt
leptons 9mo ago
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
- chrisjj 9mo ago
  s/hallucinated/fabricated/, please.
  - Terretta 9mo ago
    arguably: incorrectly guessed* in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be "confabulation", though, seems perfect: “Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.” https://en.wikipedia.org/wiki/Confabulation https://en.wikipedia.org/wiki/Confabulation * insofar as “guess” conveys an attempt to be probably in the zone
- terminalshort 9mo ago
  What medication is this? It's always given me good info.
  - leptons 9mo ago
    I'm not sure why you would think I would blurt that out on the internet. And are you sure it's giving you good info? "AI" is famously subject to hallucinations, so you may not be getting the "good info" you think you're getting. Be careful with "AI", it's not an all-seeing-all-knowing infallible oracle.
    - terminalshort 9mo ago
jeffbee 9mo ago
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
- gvedem 9mo ago
  Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
  - what 9mo ago
    What do you use instead… that doesn’t piggyback off of google search?
- benregenspan 9mo ago
  It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
- what 9mo ago
  The AI summaries clearly aren’t bad. I’m not sure what kind of weird shit you search for that you consider the summaries bad. I find them helpful and click through to the cited sources.
  - xxs 9mo ago
    ...and the cited source is AI generated video(s). There are summaries that say exactly the opposite of the correct result.
InMice 9mo ago
Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
- yakattak 9mo ago
  I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
- Loughla 9mo ago
  I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it. Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for. I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
  - lelandfe 9mo ago
    You can cure Google's AI (but not bad results) with &udm=14 https://google.com/search?q=parkas&udm=14 https://google.com/search?q=parkas&udm=14
- ajross 9mo ago
  What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me? I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone must have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had. Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
deleted 9mo ago
[deleted]
myhf 9mo ago
If an app makes a diagnosis or a recommendation based on health data, that's Software as a Medical Device (SaMD) and it opens up a world of liability. https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd https://www.fda.gov/medical-devices/digital-health-center-ex...
- deleted 9mo ago
  [deleted]
- eastbound 9mo ago
  How do you suggest to deal with Gemini? Extremely useful to understand whether something is worrying or not. Whether we like it or not, it’s a main participant to the discussion.
  - jessetemp 9mo ago
    Ideally, hold Google liable until their AI doesn’t confabulate medical advice. Realistically, sign a EULA waiving your rights because their AI confabulates medical advice
  - overfeed 9mo ago
    > How do you suggest to deal with Gemini? With robust fines based on % revenue whenever it breaks the law, would be my preference. I'm nit here to attempt solutions to Google's self-inflicted business-model challenges.
  - deleted 9mo ago
    [deleted]
  - arkh 9mo ago
    > How do you suggest to deal with Gemini? Don't. I do not ask my mechanic for medical advice, why would I ask a random output machine?
    - menaerus 9mo ago
      This "random output machine" is already in large use in medicine so why exactly not? Should I trust the young doctor fresh out of the Uni more by default or should I take advises from both of them with a grain of salt? I had failures and successes with both of them but lately I found Gemini to be extremely good at what it does.
ipython 9mo ago
... at the same time, OpenAI launches their ChatGPT Health service: https://openai.com/index/introducing-chatgpt-health/ https://openai.com/index/introducing-chatgpt-health/, marketed as "a dedicated experience in ChatGPT designed for health and wellness." So interesting to see the vastly different approaches to AI safety from all the frontier labs.
- _cs2017_ 9mo ago
  Why vastly different? Aren't they both searching various online sources for relevant information and feeding that into the LLM?
  - simianwords 9mo ago
    Different levels of capabilities. The summary feature in google uses a quick and inaccurate AI model. Were it to be a heavier model, we wouldn’t have this problem.
    - thepotatodude 9mo ago
      We would still have this problem. The heavier models make mistakes at too high a rate vs. a physician. Especially on imaging data. Real world data and patient presentations often deviate from the textbooks they are trained on. -Med student
      - simianwords 9mo ago
        That's a different class of problem. It will do just fine on text based queries spanning a few pages. Probably better than the average physician (average over all countries). I do agree that LLM's are not there yet in the image part.
Hippieblog 9mo ago
[dead]
Terretta 9mo ago
> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added. Notice how little this sentence says about whether anything is any good.
- user34283 9mo ago
  I can't speak to Health summaries, but the Weather summary on Pixel phones is not good. Like you could have a few days of -3C, for today it goes up to +5C, and the "AI Weather report" tells you it's going to be a chilly day or something. I never saw this feature provide any useful information whatsoever.
- 63stack 9mo ago
  No sane, in the loop person would believe Google reviews anything, at least not with a human. I bet they have another AI reviewing the answers, and the only time a human enters the loop is when this second AI needs to be restarted.
bjourne 9mo ago
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy. Also try "health benefits of circumcision"...
- catlikesshrimp 9mo ago
  I agree with your point. Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications. I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
  - serf 9mo ago
    the problem isn't that search engines are polluted; that's well known. The problem is that people perceive these AI responses as something greater than a search query; they view it as an objective view point that was reasoned out by some sound logical method -- and anyone that understands the operation of LLMs knows that they don't really do that, except for some very specific edge examples.
SirIsaacGluten 9mo ago
How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?
- FireBeyond 9mo ago
  Being gutted by DOGE and the Trump Administration under RFK Jr.
nickphx 9mo ago
huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol
mannykannot 9mo ago
The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.
hsuduebc2 9mo ago
They AI summary is total garbage. Probably most broken feature I saw being released in a while.
Nursie 9mo ago
But only for some highly specific searches, when what it should be doing is checking if it's any sort of medical query and keeping the hell out of it because it can't guarantee reliability. It's still baffling to me that the world's biggest search company has gone all-in on putting a known-unreliable summary at the top of its results.
chanux 9mo ago
This incessant, unchecked[1] peddling is what rids "AI" of the good name it could earn for the things it's good at. But Alas, infinite growth or nothing is the name of the game now. [1] Well, not entirely thanks to people investigating.
bsimpson 9mo ago
Tangent, but some people I know have been downloading their genomes from 23andme and asking Gemini via Antigravity to analyze it. "If you don't die of heart disease by 50, you'll probably live to be 100." I wonder how accurate it is.
- sitharus 9mo ago
  As accurate as our knowledge of genetics, which is not very outside of the identified set of pathological genes associated with hereditary disorders. Your genome is very complex and we don’t have a model of how every gene interacts with every other and how they’re affected by your environment. Geneticists are working on it, but it’s not here yet. And remember that 23andMe, Ancestry, and most other services only sequence around 1% of your genome.
  - CJefferson 9mo ago
    I'd guess it's much less accurate than that. Part of genetics is pattern matching, and last time I checked I still can't find a model that can correctly solve hard Sudokus (well, assuming you don't pick a coding model that writes a Sudoku solver.. maybe some of them are trying to do genetics by doing correct algorithms), a trivial job if you write a program that is designed to do it.
- socalgal2 9mo ago
  Are you asking for you in particular? It's certainly not accurate in general that anyone that made it to 50 is likely to live to 100. One I heard was if you make it to 80 you have a 50% chance to make it to 90. If you make it to 90 you have a 50% chance to make it to 95. From 95 to 97.5 again 50% chance. That for the general population, in a 1st world country though, not any individual.
  - bsimpson 9mo ago
    The assessment he got was "you have supergenerian genes, but also this one that's really sensitive to heart disease. If you can keep you plaque in check, you're almost invincible (to other popular causes of mortality)." It cited whatever gene it said he had that made that so.
inquirerGeneral 9mo ago
Claude just added Health Connect integration for Android. Meanwhile Copilot launched a full bot for it: "Dos and don’ts of medical AI While AI is a useful tool that can help you understand medical information, it’s important to clarify what it’s designed to do (and what it isn’t). Dos: Use AI as a reliable guide for finding doctors and understanding care options. Let AI act as an always available medical assistant that explains information clearly. Use AI as a transparent, unbiased source of clinically validated health content. Don’ts: Don’t use AI for medical diagnosis. If you’re concerned you may have a medical issue, you should seek the help of a medical professional. Don’t replace your doctor or primary care provider with an “AI doctor”. AI isn’t a doctor. You should always consult a professional before making any medical decisions. This clarity is what makes Copilot safe" https://www.microsoft.com/en-us/microsoft-copilot/for-individuals/do-more-with-ai/ai-for-daily-life/copilot-for-health-your-ai-companion-for-smarter-medical-decisions#:~:text=Dos%20and%20don%E2%80%99ts,makes%20Copilot%20safe https://www.microsoft.com/en-us/microsoft-copilot/for-indivi...
utopiah 9mo ago
Meanwhile ChatGPT health just 5 days ago https://openai.com/index/introducing-chatgpt-health/ https://openai.com/index/introducing-chatgpt-health/ because clearly they both don't give a fuck AND they are so desperate for money, any money.
n8m8 9mo ago
It took being a meme for a literal year for them to remove this… more responsibility in frontier tech, I’m begging.
thepotatodude 9mo ago
I'm telling you all this as a medical student that has used the latest and greatest models with proper prompting for the past 3 years in school: There are a ton of misses. Especially on imaging. LLMs are not ready for consumer-facing health information yet. My guess is ~ 3-5 years. Right now, I see systems implementing note writing with LLMs, which is hit or miss (but will rapidly improve). Physicians want 1:1 customization. Have someone sit with them and talk through how they like their notes/set it up so the LLMs produce notes like that. Notes need to be customized at the physician level. Also, the electronic health records any AI is trained on is loaded to the brim with borderline fraud/copy paste notes. That's going to have to be reconciled. Do we have the LLMs add "Cranial Nerves II-X intact" even though the physician did not actually assess that? The physician would have documented that... No? But then you open up the physician to liability, which is a no go for adopting software. Building a SaaS MVP that's 80% of the way there? Sure. But medicine is not an MVP you cram into a pitch deck for a VC. 80% of the way there does not cut it here, especially if we're talking about consumer facing applications. Remember, the average American reads at a 6th grade reading level. Pause and let that sink in. You're probably surrounded by college educated people like yourself. It was a big shock when I started seeing patients, even though I am the first in my family to go to college. Any consumer-facing health AI tool needs to be bulletproof!! Big Tech will not deliver us a healthcare utopia. Do not buy into their propaganda. They are leveraging post-pandemic increases in mistrust towards physicians as a springboard for half-baked solutions. Want to make $$$ doing the same thing? Do it in a different industry.
totetsu 9mo ago
"unsafe at any seed" is very cleaver subtitle for this article.

226 points by barishnamazov 9mo ago

"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them... https://deadstack.net/cluster/google-removes-ai-overviews-for-medical-queries https://deadstack.net/cluster/google-removes-ai-overviews-fo...

bandrami 9mo ago
Why does healthcare "need disruption"?
- miltonlost 9mo ago
  The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
  - zdragnar 9mo ago
    The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
    - kunai 9mo ago
      Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
  - eru 9mo ago
    Profit is a small part of overall revenue.
- wswin 9mo ago
  It's inefficient and not living to its potential
  - bandrami 9mo ago
    And "disruption" (a pretty ill-defined term) is the solution to that?
    - __loam 9mo ago
      The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
      - gradus_ad 9mo ago
        There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here. And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
        __loam 9mo ago
        We'd rather have medical bankruptcy than a functional system.
        ikr678 9mo ago
        Maybe for R&D but your outsized costs are also due to your liability environment and direct incentives against preventative healthcare.
  - plagiarist 9mo ago
    The inefficiency is the buying of yachts for billionaires.
- dyauspitr 9mo ago
  Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
  - gmueckl 9mo ago
    But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
    - eru 9mo ago
      Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance? Profit isn't even a big part of the overall revenue. > Mandate at least decent minimal coverage standards I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices. > and large insurance pools that must span age groups and risk groups. Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
      - FireBeyond 9mo ago
        > Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you). Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
- harpratap 9mo ago
  Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
  - eru 9mo ago
    I sympthatise with the argument. We should test it against real world data. Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
    - eru 9mo ago
      I just looked it up, and apparently health care costs for pets has gone up in price even faster than for humans. Pets typically don't have medical insurance, and any insurance that does exist there has a radically different regulatory regime than for humans. Since 1980 for the US: CPI has gone up by 3.16% on average per year (x4.17 in total). Human healthcare costs by 4.9% per year (x8.96 in total). And pet healthcare costs by 6.49% (or x17.87 in total).
      - manuelmoreale 9mo ago
        I suspect this has something to do with it: https://www.theatlantic.com/ideas/archive/2024/04/vet-private-equity-industry/678180/ https://www.theatlantic.com/ideas/archive/2024/04/vet-privat...
        eru 9mo ago
        That could at most explain the last few years of rising vet prices.
        bandrami 9mo ago
- sergiotapia 9mo ago
  because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
  - sarchertech 9mo ago
    Physician compensation is around 9% of healthcare spending. The number of non-physician providers (NPs, PAs and specialists like physical therapists and podiatrists) has also exploded over the last 20 years. We have far more overall providers per capita than we did 20 years ago. Lack of providers isn’t what’s driving up costs.
    - derbOac 9mo ago
      I don't think physician compensation per se is a good metric for capturing the effect of lack of providers, because some of the increased costs are due to the bottlenecks in the services per se, in terms of procedure costs and types of procedures offered. I also don't think the number of providers per se under the current regime, without deregulation or reregulation of practice boundaries, is representative of what would happen if there were changes in those boundaries. Adding more optometrists 5 years ago isn't the same as changing what they're allowed to do. It also doesn't address what cost increases would have been without an increase in the number of providers. 9% might also seem pretty big to me if it's out of all spending and doesn't include other provider compensation? What if overall healthcare costs went down, but physician compensation stayed the same? Would that then be a problem because it was an increased proportion of the total costs — fat left to be trimmed, so to speak? There are many problems that don't have anything to do with providers per se, but I also don't think you can glean much by extrapolating to more of the same, especially compensation per se.
      - thepotatodude 9mo ago
        9% is nothing when you consider that these people provide the value. With no physicians, there is no healthcare.
      - 9mo ago
- seanmcdirmid 9mo ago
  Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
  - scarab92 9mo ago
    Society pretends that human doctors are better than they really are, and AI is worse than it really is. It's the self-driving cars debate all over again.
    - thepotatodude 9mo ago
      It's really just not there yet. I've been in medical school for >3 years now and have been using the latest models with good prompting. They have gotten much better, but I still see misses that my classmates would easily catch. This is not acceptable in healthcare. It's certainly not getting 100% on all my assignments, which are a step below the complexity of real-world clinical practice. Before medical school, I was not so sure of the quality of your average doc. Now having spent a year in clinical practice across various settings, I am extremely reassured. I can say with certainty that a US trained doctor is miles ahead of AI right now. The system sucks really bad though and forces physicians to churn patients, giving the impression that physicians don't pay attention/don't care/etc.
  - bandrami 9mo ago
    I mean, if we're talking Christensenian disruption, that happens in neglected markets rather than currently dysfunctional ones. There's no shortage of actors wronging money out of health care so there's not a disruptable space per se.
    - omnicognate 9mo ago
      > wronging Not sure if this is a typo (of wringing) or a pun, but it's apt either way.

xnx 9mo ago

Ars rips of this original reporting, but makes it worse by leaving out the word "some" from the title. ‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk: https://www.theguardian.com/technology/2026/jan/11/google-ai-overviews-health-guardian-investigation https://www.theguardian.com/technology/2026/jan/11/google-ai...

miltonlost 9mo ago
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk" Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"

TwoNineA 9mo ago

[flagged]

moss_dog 9mo ago
What's that?
- BloodyIron 9mo ago
  lol you must be fresh to the internet

jnamaya 9mo ago

chatGPT told me, I am the healthiest guy in the world, and I believe it

deleted 9mo ago
[deleted]
xxs 9mo ago
If it's any help women appreciate confidence; Yet, the article is not about chatgpt

leptons 9mo ago

Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.

chrisjj 9mo ago
s/hallucinated/fabricated/, please.
- Terretta 9mo ago
  arguably: incorrectly guessed* in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be "confabulation", though, seems perfect: “Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.” https://en.wikipedia.org/wiki/Confabulation https://en.wikipedia.org/wiki/Confabulation * insofar as “guess” conveys an attempt to be probably in the zone
terminalshort 9mo ago
What medication is this? It's always given me good info.
- leptons 9mo ago
  I'm not sure why you would think I would blurt that out on the internet. And are you sure it's giving you good info? "AI" is famously subject to hallucinations, so you may not be getting the "good info" you think you're getting. Be careful with "AI", it's not an all-seeing-all-knowing infallible oracle.
  - terminalshort 9mo ago

jeffbee 9mo ago

Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.

gvedem 9mo ago
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
- what 9mo ago
  What do you use instead… that doesn’t piggyback off of google search?
benregenspan 9mo ago
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
what 9mo ago
The AI summaries clearly aren’t bad. I’m not sure what kind of weird shit you search for that you consider the summaries bad. I find them helpful and click through to the cited sources.
- xxs 9mo ago
  ...and the cited source is AI generated video(s). There are summaries that say exactly the opposite of the correct result.

InMice 9mo ago

Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.

yakattak 9mo ago
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
Loughla 9mo ago
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it. Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for. I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
- lelandfe 9mo ago
  You can cure Google's AI (but not bad results) with &udm=14 https://google.com/search?q=parkas&udm=14 https://google.com/search?q=parkas&udm=14
ajross 9mo ago
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me? I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone must have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had. Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".

deleted 9mo ago

[deleted]

myhf 9mo ago

If an app makes a diagnosis or a recommendation based on health data, that's Software as a Medical Device (SaMD) and it opens up a world of liability. https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd https://www.fda.gov/medical-devices/digital-health-center-ex...

deleted 9mo ago
[deleted]
eastbound 9mo ago
How do you suggest to deal with Gemini? Extremely useful to understand whether something is worrying or not. Whether we like it or not, it’s a main participant to the discussion.
- jessetemp 9mo ago
  Ideally, hold Google liable until their AI doesn’t confabulate medical advice. Realistically, sign a EULA waiving your rights because their AI confabulates medical advice
- overfeed 9mo ago
  > How do you suggest to deal with Gemini? With robust fines based on % revenue whenever it breaks the law, would be my preference. I'm nit here to attempt solutions to Google's self-inflicted business-model challenges.
- deleted 9mo ago
  [deleted]
- arkh 9mo ago
  > How do you suggest to deal with Gemini? Don't. I do not ask my mechanic for medical advice, why would I ask a random output machine?
  - menaerus 9mo ago
    This "random output machine" is already in large use in medicine so why exactly not? Should I trust the young doctor fresh out of the Uni more by default or should I take advises from both of them with a grain of salt? I had failures and successes with both of them but lately I found Gemini to be extremely good at what it does.

ipython 9mo ago

... at the same time, OpenAI launches their ChatGPT Health service: https://openai.com/index/introducing-chatgpt-health/ https://openai.com/index/introducing-chatgpt-health/, marketed as "a dedicated experience in ChatGPT designed for health and wellness." So interesting to see the vastly different approaches to AI safety from all the frontier labs.

_cs2017_ 9mo ago
Why vastly different? Aren't they both searching various online sources for relevant information and feeding that into the LLM?
- simianwords 9mo ago
  Different levels of capabilities. The summary feature in google uses a quick and inaccurate AI model. Were it to be a heavier model, we wouldn’t have this problem.
  - thepotatodude 9mo ago
    We would still have this problem. The heavier models make mistakes at too high a rate vs. a physician. Especially on imaging data. Real world data and patient presentations often deviate from the textbooks they are trained on. -Med student
    - simianwords 9mo ago
      That's a different class of problem. It will do just fine on text based queries spanning a few pages. Probably better than the average physician (average over all countries). I do agree that LLM's are not there yet in the image part.

Hippieblog 9mo ago

[dead]

Terretta 9mo ago

> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added. Notice how little this sentence says about whether anything is any good.

user34283 9mo ago
I can't speak to Health summaries, but the Weather summary on Pixel phones is not good. Like you could have a few days of -3C, for today it goes up to +5C, and the "AI Weather report" tells you it's going to be a chilly day or something. I never saw this feature provide any useful information whatsoever.
63stack 9mo ago
No sane, in the loop person would believe Google reviews anything, at least not with a human. I bet they have another AI reviewing the answers, and the only time a human enters the loop is when this second AI needs to be restarted.

bjourne 9mo ago

Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy. Also try "health benefits of circumcision"...

catlikesshrimp 9mo ago
I agree with your point. Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications. I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
- serf 9mo ago
  the problem isn't that search engines are polluted; that's well known. The problem is that people perceive these AI responses as something greater than a search query; they view it as an objective view point that was reasoned out by some sound logical method -- and anyone that understands the operation of LLMs knows that they don't really do that, except for some very specific edge examples.

SirIsaacGluten 9mo ago

How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?

FireBeyond 9mo ago
Being gutted by DOGE and the Trump Administration under RFK Jr.

nickphx 9mo ago

huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol

mannykannot 9mo ago

The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.

hsuduebc2 9mo ago

They AI summary is total garbage. Probably most broken feature I saw being released in a while.

Nursie 9mo ago

But only for some highly specific searches, when what it should be doing is checking if it's any sort of medical query and keeping the hell out of it because it can't guarantee reliability. It's still baffling to me that the world's biggest search company has gone all-in on putting a known-unreliable summary at the top of its results.

chanux 9mo ago

This incessant, unchecked[1] peddling is what rids "AI" of the good name it could earn for the things it's good at. But Alas, infinite growth or nothing is the name of the game now. [1] Well, not entirely thanks to people investigating.

bsimpson 9mo ago

Tangent, but some people I know have been downloading their genomes from 23andme and asking Gemini via Antigravity to analyze it. "If you don't die of heart disease by 50, you'll probably live to be 100." I wonder how accurate it is.

sitharus 9mo ago
As accurate as our knowledge of genetics, which is not very outside of the identified set of pathological genes associated with hereditary disorders. Your genome is very complex and we don’t have a model of how every gene interacts with every other and how they’re affected by your environment. Geneticists are working on it, but it’s not here yet. And remember that 23andMe, Ancestry, and most other services only sequence around 1% of your genome.
- CJefferson 9mo ago
  I'd guess it's much less accurate than that. Part of genetics is pattern matching, and last time I checked I still can't find a model that can correctly solve hard Sudokus (well, assuming you don't pick a coding model that writes a Sudoku solver.. maybe some of them are trying to do genetics by doing correct algorithms), a trivial job if you write a program that is designed to do it.
socalgal2 9mo ago
Are you asking for you in particular? It's certainly not accurate in general that anyone that made it to 50 is likely to live to 100. One I heard was if you make it to 80 you have a 50% chance to make it to 90. If you make it to 90 you have a 50% chance to make it to 95. From 95 to 97.5 again 50% chance. That for the general population, in a 1st world country though, not any individual.
- bsimpson 9mo ago
  The assessment he got was "you have supergenerian genes, but also this one that's really sensitive to heart disease. If you can keep you plaque in check, you're almost invincible (to other popular causes of mortality)." It cited whatever gene it said he had that made that so.

inquirerGeneral 9mo ago

Claude just added Health Connect integration for Android. Meanwhile Copilot launched a full bot for it: "Dos and don’ts of medical AI While AI is a useful tool that can help you understand medical information, it’s important to clarify what it’s designed to do (and what it isn’t). Dos: Use AI as a reliable guide for finding doctors and understanding care options. Let AI act as an always available medical assistant that explains information clearly. Use AI as a transparent, unbiased source of clinically validated health content. Don’ts: Don’t use AI for medical diagnosis. If you’re concerned you may have a medical issue, you should seek the help of a medical professional. Don’t replace your doctor or primary care provider with an “AI doctor”. AI isn’t a doctor. You should always consult a professional before making any medical decisions. This clarity is what makes Copilot safe" https://www.microsoft.com/en-us/microsoft-copilot/for-individuals/do-more-with-ai/ai-for-daily-life/copilot-for-health-your-ai-companion-for-smarter-medical-decisions#:~:text=Dos%20and%20don%E2%80%99ts,makes%20Copilot%20safe https://www.microsoft.com/en-us/microsoft-copilot/for-indivi...

utopiah 9mo ago

Meanwhile ChatGPT health just 5 days ago https://openai.com/index/introducing-chatgpt-health/ https://openai.com/index/introducing-chatgpt-health/ because clearly they both don't give a fuck AND they are so desperate for money, any money.

n8m8 9mo ago

It took being a meme for a literal year for them to remove this… more responsibility in frontier tech, I’m begging.

thepotatodude 9mo ago

I'm telling you all this as a medical student that has used the latest and greatest models with proper prompting for the past 3 years in school: There are a ton of misses. Especially on imaging. LLMs are not ready for consumer-facing health information yet. My guess is ~ 3-5 years. Right now, I see systems implementing note writing with LLMs, which is hit or miss (but will rapidly improve). Physicians want 1:1 customization. Have someone sit with them and talk through how they like their notes/set it up so the LLMs produce notes like that. Notes need to be customized at the physician level. Also, the electronic health records any AI is trained on is loaded to the brim with borderline fraud/copy paste notes. That's going to have to be reconciled. Do we have the LLMs add "Cranial Nerves II-X intact" even though the physician did not actually assess that? The physician would have documented that... No? But then you open up the physician to liability, which is a no go for adopting software. Building a SaaS MVP that's 80% of the way there? Sure. But medicine is not an MVP you cram into a pitch deck for a VC. 80% of the way there does not cut it here, especially if we're talking about consumer facing applications. Remember, the average American reads at a 6th grade reading level. Pause and let that sink in. You're probably surrounded by college educated people like yourself. It was a big shock when I started seeing patients, even though I am the first in my family to go to college. Any consumer-facing health AI tool needs to be bulletproof!! Big Tech will not deliver us a healthcare utopia. Do not buy into their propaganda. They are leveraging post-pandemic increases in mistrust towards physicians as a springboard for half-baked solutions. Want to make $$$ doing the same thing? Do it in a different industry.

totetsu 9mo ago

"unsafe at any seed" is very cleaver subtitle for this article.