10 ms·
I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor
- freedomben 9mo ago> Despite having access to my weight, blood pressure and cholesterol, ChatGPT based much of its negative assessment on an Apple Watch measurement known as VO2 max, the maximum amount of oxygen your body can consume during exercise. Apple says it collects an “estimate” of VO2 max, but the real thing requires a treadmill and a mask. Apple says its cardio fitness measures have been validated, but independent researchers have found those estimates can run low — by an average of 13 percent. There's plenty of blame to go around for everyone, but at least for some of it (such as the above) I think the blame more rests on Apple for falsely representing the quality of their product (and TFA seems pretty clearly to be blasting OpenAI for this, not others like Apple). What would you expect the behavior of the AI to be? Should it always assume bad data or potentially bad data? If so, that seems like it would defeat the point of having data at all as you could never draw any conclusions from it. Even disregarding statistical outliers, it's not at all clear what part of the data is "good" vs "unrealiable" especially when the company that collected that data claims that it's good data.
- hmokiguess 9mo agoI have been sitting and waiting for the day these trackers get exposed as just another health fad that is optimized to deliver shareholder value and not serious enough for medical grade applications
- NoPicklez 9mo agoI don't see how they are considered a health fad, they're extremely useful and accurate enough. There are plenty of studies and real world data showing Garmin VO2Max readings being accurate to 1-2 points different to a real world test. There is this constant debate about how accurately VO2max is measured and its highly dependent on actually doing exercise to determine your VO2max using your watch. But yes if you want a lab/medically precise measure you need to do it a test that measures your actual oxygen uptake.
- aeonfox 9mo ago> I think the blame more rests on Apple for falsely representing the quality of their product There was plenty of other concerning stuff in that article. And from a quick read it wasn't suggested or implied the VO2 max issue was the deciding factor for the original F score the author received. The article did suggest many times over the ChatGPT is really not equipped for the task of health diagnosis. > There was another problem I discovered over time: When I tried asking the same heart longevity-grade question again, suddenly my score went up to a C. I asked again and again, watching the score swing between an F and a B.
- brandonb 9mo agoThe lack of self-consistency does seem like a sign of a deeper issue with reliability. In most fields of machine learning robustness to noise is something you need to "bake in" (often through data augmentation using knowledge of the domain) rather than get for free in training.
- freedomben 9mo ago> There was plenty of other concerning stuff in that article. Yeah for sure, I probably didn't make it clear enough but I do fault OpenAI for this as much as or maybe more than Apple. I didn't think that needed to be stressed since the article is already blasting them for it and I don't disagree with most of that criticism of OpenAI.
- brandonb 9mo agoFWIW, Apple has published validation data showing the Apple Watch's estimate is within 1.2 ml/kg/min of a lab-measured Vo2Max. Behind the scenes, it's using a pretty cool algorithm that combines deep learning with physiological ODEs: https://www.empirical.health/blog/how-apple-watch-cardio-fitness-vo2max-works https://www.empirical.health/blog/how-apple-watch-cardio-fit...
- ignoramous 9mo agoThe paper itself: https://www.apple.com/healthcare/docs/site/Using_Apple_Watch_to_Estimate_Cardio_Fitness_with_VO2_max.pdf https://www.apple.com/healthcare/docs/site/Using_Apple_Watch... Seems like Apple's 95% accuracy estimate for VO2 max holds up. Thirty participants wore an Apple Watch for 5-10 days to generate a VO2 max estimate. Subsequently, they underwent a maximal exercise treadmill test in accordance with the modified Åstrand protocol. The agreement between measurements from Apple Watch and indirect calorimetry was assessed using Bland-Altman analysis, mean absolute percentage error (MAPE), and mean absolute error (MAE). Overall, Apple Watch underestimated VO2 max, with a mean difference of 6.07 mL/kg/min (95% CI 3.77–8.38). Limits of agreement indicated variability between measurement methods (lower -6.11 mL/kg/min; upper 18.26 mL/kg/min). MAPE was calculated as 13.31% (95% CI 10.01–16.61), and MAE was 6.92 mL/kg/min (95% CI 4.89–8.94). These findings indicate that Apple Watch VO2 max estimates require further refinement prior to clinical implementation. However, further consideration of Apple Watch as an alternative to conventional VO2 max prediction from submaximal exercise is warranted, given its practical utility. https://pmc.ncbi.nlm.nih.gov/articles/PMC12080799/ https://pmc.ncbi.nlm.nih.gov/articles/PMC12080799/
- miltonlost 9mo ago> What would you expect the behavior of the AI to be? Should it always assume bad data or potentially bad data? If so, that seems like it would defeat the point of having data at all as you could never draw any conclusions from it. Well, I would expect the AI to provide the same response as a real doctor did from the same information. Which the article went over the doctors were able to. I also would expect the AI to provide the same answer every time to the same data unlike what it did (from F to B over multiple attempts in the article) OpenAI is entirely to blame here when they are putting out faulty products, (hallucinations even on accurate data are a fault of them).
- jdub 9mo agoWhy do you have those expectations?
- AndrewKemendo 9mo ago> Should it always assume bad data or potentially bad data? If so, that seems like it would defeat the point of having data at all as you could never draw any conclusions from it. Yes. You, and every other reasoning system, should always challenge the data and assume it’s biased at a minimum. This is better described as “critical thinking” in its formal form. You could also call it skepticism. That impossibility of drawing conclusions assumes there’s a correct answer and is called the “problem of induction.” I promise you a machine is better at avoiding it than a human. Many people freeze up or fail with too much data - put someone with no experience in front of 500 ppl to give a speech if you want to watch this live.
- freedomben 9mo agoI mostly agree with you, but I think it's important to consider what you're doing with the data. If we're doing rigorous science, or making life-or-death decisions on it, I would 100% agree. But if we're an AI chatbot trying to offer some insight, with a big disclaimer that "these results might be wrong, talk to your doctor" then I think that's quite overkill. The end result would be no (potential) insight at all and no chance for ever improving since we'll likely never get a to a point where we could fully trust the data. Not even the best medical labs are always perfect.
- jayd16 9mo agoWell if it doesn't know the quality of the data and especially if it would be dangerous to guess then it should probably say it doesn't have an answer.
- freedomben 9mo agoI don't disagree, but that reinforces my point above I think. If AI has to assume the data is of poor quality, then there's no point in even trying to analyze it. The options are basically: 1. Trust the source of the data to be honest about it's quality Or 2. Distrust the source Approach number 2 basically means we can never do any analysis on it. Personally I'd rather have a product that might be wrong than none at all, but that's a personal preference.
- dfajgljsldkjag 9mo agoThe author is a healthy person but the computer program still gave him a failing grade of F. It is irresponsible for these companies to release broken tools that can cause so much fear in real people. They are treating serious medical advice like it is just a video game or a toy. Real users should not be the ones testing these dangerous products.
- dylan604 9mo agoWhat LLM should the LLM turn to ask if what the user is asking is safe for the first LLM to answer?
- nomel 9mo ago> It is irresponsible for these companies I would claim that ignoring the "ChatGPT is AI and can make mistakes. Check important info." text, right under the query they type in client, is clearly more irresponsible. I think that a disclaimer like that is the most useful and reasonable approach for AI. "Here's a tool, and it's sometimes wrong." means the public can have access to LLMs and AI. The alternative, that you seem to be suggesting (correct me if I'm wrong), means the public can't have access to an LLM until they are near perfect, which means the public can't ever have access to an LLM, or any AI. What do you see as a reasonable approach to letting the public access these imperfect models? Training? Popups/agreement after every question "I understand this might be BS"? What's the threshold for quality of information where it's no longer considered "broken"? Is that threshold as good as or better than humans/news orgs/doctors/etc?
- zdragnar 9mo ago> Popups/agreement after every question "I understand this might be BS"? Considering the number of people who take LLM responses as authoritative Truth, that wouldn't be the worst thing in the world.
- ytoawwhra92 9mo agoWhy are you assuming that the general public ought to have access to imperfect tools? I live in a place where getting a blood test requires a referral from a doctor, who is also required to discuss the results with you.
- brandonb 9mo agoWe trained a foundation model specifically for wearable data: https://www.empirical.health/blog/wearable-foundation-model-jets/ https://www.empirical.health/blog/wearable-foundation-model-... The basic idea was to adapt JEPA (Yann LeCun's Joint-Embedding Predictive Architecture) to multivariate time series, in order to learn a latent space of human health from purely unlabeled data. Then, we tested the model using supervised fine tuning and evaluation on on a bunch of downstream tasks, such as predicting a diagnosis of hypertension (~87% accuracy). In theory, this model could be also aligned to the latent space of an LLM--similar to how CLIP aligns a vision model to an LLM. IMO, this shows that accuracy in consumer health will require specialized models alongside standard LLMs.
- creatonez 9mo agoChatGPT Health is a completely wreckless and dangerous product, they should be sued into oblivion for even naming it "health".
- orionsbelt 9mo agoChatGPT has done more for my health than any doctor. Truly.
- haldujai 9mo agoHow so?
- theshrike79 9mo agoChatGPT will actually look at your whole medical history, listen to you, think and check multiple different options before making a decision. You can spend hours chatting with it back and forth. An average human doctor has maybe 15 minutes allotted to getting to know you, analyse and determine a course of action. Which is usually "take some ibuprofen and let's see if it goes away". Then you go again in two weeks with the same thing, it's a different doctor and the context has been reset unless you do an info dump from the previous visits and try not to forget anything. And if you infodump too much or use actual medical diagnosis terms, the Dr gets defensive because you're stepping on THEIR area of expertise and will start pushing back even from the obvious just because they can.
- eur0pa 9mo agoA LLM neither "listens" nor "thinks"
- theshrike79 9mo agoFor the sake of fluid writing, I did use anthropomorphic verbs. What would you prefer instead?
- 9mo ago
- CqtGLRGcukpy 9mo agoOriginal article can be read at https://www.washingtonpost.com/technology/2026/01/26/chatgpt-health-apple/ https://www.washingtonpost.com/technology/2026/01/26/chatgpt.... Paywall-free version at https://archive.ph/k4Rxt https://archive.ph/k4Rxt
- maxdo 9mo agoTypical Western coverage: “How dare they call me unhealthy.” In reality, the doctor said it needs further investigation and that some data isn’t great. They didn’t say “unhealthy”; they said “needs more investigation.” What’s wrong with that? Is the real issue just a bruised Western ego?
- smileysteve 9mo agoAlt; typical western coverage. Has completely ignored other journalists publishing of the plight fitness bands have caused in doctors "am I getting a cold, my watch/ring says I'm getting a cold, give me antibiotics now" Or how vo2 max is hard to measure, or how not wearing a wearable or wearing it loose changes results, to, I gave an llm a range to rate without really giving it context of what I want the range to really represent or the methods of gathering data. Tldr; author bought everything, read nothing, complained to an expensive professional, and now hopes that we read his article.
- wawayanda 9mo agoA year or so ago, I fed my wife's blood work results into chatgpt and it came back with a terrifying diagnosis. Even after a lot of back and forth it stuck to its guns. We went to a specialist who performed some additional tests and explained that the condition cannot be diagnosed with just the original blood work and said that she did not have the condition. The whole thing was a borderline traumatic ordeal that I'm still pretty pissed about.
- daveguy 9mo agoPlease keep telling your story. This is the kind of shit that medical science has been dealing with for at least a century. When evaluating testing procedures false positives can have serious consequences. A test that's positive every time will catch every single true positive, but it's also worthless. These LLMs don't have a goddamn clue about it. There should be consequences for these garbage fires giving medical advice.
- maerF0x0 9mo agoPart of the issue is taking it's output as conclusion rather than as a signal / lead. I would never let an LLM make an amputate or not decision, but it could convince me to go talk with an expert who sees me in person and takes a holistic view.
- irjustin 9mo agoIsn't it two sides to the same coin? You should be happy about it that it's not the thing specifically when the signs pointed towards it being "the thing"?
- anonzzzies 9mo agoApple watch told me, based on vo2 max, that i'm almost dead, all the time. I went to the doctor, did a real test and it was complete nonsense. I had the watch replaced 3 times but same results, so I returned it and will not try again. Scaring people with stuff you cannot actually shut off (at least you couldn't before) is not great.
- chrisfosterelli 9mo agoHealth metrics are absolutely tarnished by a lack of proper context. Unsurprisingly, it turns out that you can't reliably take a concept as broad as health and reduce it to a number. We see the same arguments over and over with body fat percentages, vo2 max estimates, BMI, lactate thresholds, resting heart rate, HRV, and more. These are all useful metrics, but it's important to consider them in the proper context that each of them deserve. This article gave an LLM a bunch of health metrics and then asked it to reduce it to a single score, didn't tell us any of the actual metric values, and then compared that to a doctor's opinion. Why anyone would expect these to align is beyond my understanding. The most obvious thing that jumps out to me is that I've noticed doctors generally, for better or worse, consider "health" much differently than the fitness community does. It's different toolsets and different goals. If this person's VO2 max estimate was under 30, that's objectively a poor VO2 max by most standards, and an LLM trained on the internet's entire repository of fitness discussion is likely going to give this person a bad score in terms of cardio fitness. But a doctor who sees a person come in who isn't complaining about anything in particular, moves around fine, doesn't have risk factors like age or family history, and has good metrics on a blood test is probably going to say they're in fine cardio health regardless of what their wearable says. I'd go so far to say this is probably the case for most people. Your average person is in really poor fitness-shape but just fine health-shape.
- colechristensen 9mo ago>I'd go so far to say this is probably the case for most people. Your average person is in really poor fitness-shape but just fine health-shape. Modern medicine has failed to move into the era of subtlety and small problems and many people suffer as a result. Fitness nerds and general non-scientists fill the gap poorly so we get a ton of guessing and anecdotal evidence and likely a whole lot of bad advice. Doctors won't say there's a problem until you're SICK and usually pretty late in the process when there's not a lot of room to make improvements. At the same time, doctors won't do anything if you're 5% off optimal, but they'll happily give you a medicine that improves one symptom that's 50% off optimal that comes along with 10 side effects. Although unless you're dying or have something really straightforward wrong with you, doctors don't do much at all besides giving you a sedative and or a stimulant. Doctors don't know what to do with small problems because they're barely studied and the people who DO try to do something don't do it scientifically.
- seemaze 9mo agoI can't wait until it starts recommending signing me up for an OpenAI personalized multi-vitamin® supscription
- meindnoch 9mo ago"You're absolutely right! I was mistaken about mercury and lead being essential minerals, and adding them to your supplements. Sorry about that!"
- seemaze 9mo agoI was casually browsing for a heath monitor when I came across Ultrahuman Blood Vision - one of the key features being AI Clinical Summary with Supplement Report.. it seems my sarcasm was late to the party.
- elzbardico 9mo agoA simple understanding of transformers should be enough to make someone see that using an LLM to analyze multi-variate time series data is a really stupid endeavor.
- nprateem 9mo agoIt should be obvious to even the most dim-witted idiot with a PhD in statistics and AI
- elzbardico 9mo agoYou only need this if you are a researcher. Undergraduate knowledge of Calculus and Linear Algebra is more than enough to have quite a good understanding of ML in general, and LLMs in particular. Maybe a very small bit of Information Theory (a couple of Shannon's papers are enough) and some classical books on Natural Language Processing from the late 90s and early 2000 so you have an idea of what Language Models are outside the modern Deep Learning driven approach.
- elzbardico 9mo agoLLMs are not a mythical universal machine learning model that you can feed any input and have it magically do the same thing a specialized ML model could do. You can't feed an LLM years of time-series meteorological data, and expect it to work as a specialized weather model, you can't feed it years of medical time-series and expect it to work as a model specifically trained, and validated on this specific kind of data. An LLM generates a stream of tokens. You feed it a giant set of CSVs, if it was not RL'd to do something useful with it, it will just try to make whatever sense of it and generate something that will most probably have no strong numerical relationship to your data, it will simulate an analysis, it won't do it. You may have a giant context windows, but attention is sparse, the attention mechanism doesn't see your whole data at the same time, it can do some simple comparisons, like figuring out that if I say my current pressure is 210X180 I should call an ER immediately. But once I send it a time-series of my twice a day blood-pressure measurements for the last 10 years, it can't make any real sense of it. Indeed, it would have been better for the author to ask the LLM to generate a python notebook to do some data analysis on it, and then run the notebook and share the result with the doctor.
- rfw300 9mo agoThis is true as a technical matter, but this isn't a technical blog post! It's a consumer review, and when companies ship consumer products, the people who use them can't be expected to understand failure modes that are not clearly communicated to them. If OpenAI wants regular people to dump their data into ChatGPT for Health, the onus is on them to make it reliable.
- themafia 9mo ago> the onus is on them to make it reliable. That is not a plausible outcome given the current technology or of any of OpenAI's demonstrated capabilities. "If Bob's Hacksaw Surgery Center wants to stay in business they have to stop killing patients!" Perhaps we should just stop him before it goes too far?
- vineyardmike 9mo ago
- stego-tech 9mo agoThis is not remotely surprising. Look, AI Healthbros, I'll tell you quite clearly what I want from your statistical pattern analyzers, and you don't even have to pay me for the idea (though I wouldn't say no to a home or Enterprise IT gig at your startup): I want an AI/ML tool to not merely analyze my medical info (ON DEVICE, no cloud sharing kthx), but also extrapolate patterns involving weather, location, screen time, and other "non-health" data. Do I record taking tylenol when the barometric pressure drops? Start alerting me ahead of time so I can try to avoid a headache. Does my screen time correlate to immediately decreased sleep scores? Send me a push notification or webhook I can act upon/script off of, like locking me out of my device for the night or dimming my lights. Am I recording higher-intensity workouts in colder temperatures or inclement weather? Start tracking those metrics and maybe keep better track of balance readings during those events for improved mobility issue detection. Got an app where I track cannabis use or alcohol consumption? Tie that to my mental health journal or biological readings to identify red flags or concerns about misuse. Stop trying to replace people like my medical care team, and instead equip them with better insights and datasets they can more quickly act upon. "Subject has been reporting more negative moods in his mental health journal, an uptick in alcohol consumption above his baseline, and inconsistent cannabis use compared to prior patterns" equips the care team with a quick, verifiable blurb from larger datasets that can accelerate care and improve patient outcomes - without the hallucinations of generative AI.
- siliconc0w 9mo agoThe problem is that false positives can be incredibly expensive in money, time, pain, and anxiety. Most people cannot afford (and healthcare system cannot handle) thousands of dollars in tests to disprove every AI hunch. And tests are rarely consequence free. This is effectively a negative externality of these AI health products and society is picking up the tab.
- mr_toad 9mo agoThis is why certain types of cancer tests are usually only performed on people over a certain age. If you test young people the false positives outnumber the true positives.
- jdub 9mo agoWhy do people even begin to believe that a large language model can usefully understand and interpret health data? Sure, LLM companies and proponents bear responsibility for the positioning of LLM tools, and particularly their presentation as chat bots. But from a systems point of view, it's hard to ignore the inequity and inconvenience of the US health system driving people to unrealistic alternatives. (I wonder if anyone's gathering comparable stats on "Doctor LLM" interactions in different countries... there were some interesting ones that showed how "Doctor Google" was more of a problem in the US than elsewhere.)
- djoldman 9mo agoI'm less interested in what "grade" the AI gave and much more interested in what therapy or remedy it would have suggested. That's curiously lacking here.
- daft_pink 9mo agothe problem with ai is that it isn’t good at recognizing red flags in data. i used it to find red flags in a financial report and it finds red flags in virtually every financial report it lays eyes on.
- evolighting 9mo agoHealth data, medical records, even research data, is very scarce in the public domain. This is not just due to so-called privacy concerns, but because such data could have generated “value” (and been sold at a good price) long before the emergence of large language models.
- ThundeChile 9mo agoI think it's quite alarming that people don't even think about the privacy when sending their health data to corporations which make a large percentage of their revenue selling the data onwards (or using it to things you didn't mean them to).
- cameldrv 9mo agoI dunno, if the Apple Watch said he had a vo2max of 30, that probably means he can’t run a mile in less than 12 minutes or so. He’s probably not at all healthy…
- smcl 9mo agoApple Watch is pretty poor at estimating VO2 max and it seems to be more correlated with how often you record exercises with said watch than with your actual health. For example I watched mine climb slowly as I prepared for my football season (beyond 50), then after the season started I I ended up playing and training just as frequently but without wearing the watch. After a few weeks (of me training and playing hard) during my next run it recorded me having a sharp decline in VO2 max (43-44ish iirc). When I started wearing it during training - you're not permitted during matches - it recorded me having a slow return to condition, without any changes to my routine. That said if it's showing someone as having 30 I don't imagine they're going to be in spectacular condition
- eleveriven 9mo agoThis is really more of an "utdoor run while wearing the watch" proxy than a true fitness measure
- port11 9mo agoI really don’t know whether to trust that specific measurement. When I was a very active runner and doing intervals to improve per-km time, my VO2max went from 38 to 42. I decided to do a professional VO2max test and got a 46. Now, 2 years later, I don’t run due to injury and a kid, and it’s resting at 34. For reference, when I went to the gym almost everyday and ran once or twice a week, the value was 32. I don’t get much utility out of it, even looking at the trends. Not sure what Apple is doing behind the scenes to get the score.
- smcl 9mo agoYeah so I know it's meant to be an estimate, but my experience of it is kinda fucky. I would really love to swap watches with an Olympic athlete (idk if they'd bother with an Apple Watch but bear with me!) and run 10k to see what the VO2 max reading for that exercise was. As I said, I think to me it's some estimate that heavily involves some "average of last N readings from the Apple VO2 max calc" function so even if you time travelled and gave it to Eilish McColgan or Mo Farah they'd be like "ehhh you had quite a good run, fatty - you jumped from 44.3 to 45" I'm not that bothered of course. For me it's just a fun metric I can attempt to optimise when training.
- sinuhe69 9mo agoMy general take on any AI/ML in medicine is that without a proper clinical validation, they are not worth to try. Also, AI Snake Oil is worth reading.
- rubatuga 9mo agoClinical validation, proper calibration, ethnic and community and population variants, questioning technique and more ...
- joelthelion 9mo agoExactly. There's a lot of potential, but it needs to be done right, otherwise it is worse than useless.
- gizmodo59 9mo agoFor every sensational article of AI was useless, there is plenty of examples where using ChatGPT to find out what else could be happening and then having a conversation with doctor has helped many that I know of anecdotally and many such reports online as well. At the end of the day, it’s yet another tool that people can use to help their lives. They have to use their brain. The culture of seeing doctor as a god doesn’t hold up anymore. So many people have had bad experiences when the entire health care industry at least in US is primarily a business than helping society get healthy.
- gizajob 9mo agoHard to tell who is stupider, the writer or ChatGPT.
- astura 9mo agoThe writer makes money from pumping out shitty click bait articles.
- FeteCommuniste 9mo agoI don't see how it's "shitty." It portrays a usage of ChatGPT that I imagine is becoming pretty typical. People are treating "AI" as an oracle. The situation isn't helped by corporate heads and LLM boosters blathering on about how AI is soon going to replace most of the workforce, boost productivity by a gazillion percent, and cure cancer.
- Barathkanna 9mo agoTLDR: AI didn’t diagnose anything, it turned years of messy health data into clear trends. That helped the author ask better questions and have a more useful conversation with their doctor, which is the real value here.
- alpineman 9mo agoMy wife is a doctor and there is a general trend at the moment of everyone thinking their intelligence in one area (say programming) carries over into other areas such as medicine, particularly with new tools such as ChatGPT. Imagine if as a dev someone came to you and told you everything that is wrong with your tech stack because they copy pasted some console errors into ChatGPT. There's a reason doctors need to spend almost a decade in training to parse this kind of info. If you do the above then please do it with respect for their profession.
- tripledry 9mo agoI'm reminded of an effect called Gell-Mann Amnesia. When reading news stories on topics you know well, you notice inaccuracies or poor reporting - but then immediately forget that lesson when reading the next article on a topic you are not familiar with. It's very similar to what happens with AI.
- FeteCommuniste 9mo ago> My wife is a doctor and there is a general trend at the moment of everyone thinking their intelligence in one area (say programming) carries over into other areas such as medicine, particularly with new tools such as ChatGPT. My wife is a lawyer and sees the same thing at her job. People "writing" briefs or doing legal "research" with GPT and then insisting that their document must be right because the magic AI box produced it.
- mr_toad 9mo ago> general trend at the moment “A little knowledge is a dangerous thing” is not new, it’s a quote/observation that goes back hundreds of years. > Imagine if as a dev someone came to you and told you everything that is wrong with your tech stack because they copy pasted some console errors into ChatGPT. You mean the PHB? They don’t need ChatGPT for that, they can cite Gartner.
- Aachen 9mo ago> I let ChatGPT analyze a decade of my Apple Watch data, then I called my doctor ... and you won't believe what happened next! Can we do away with the clickbait from MSN? The article is about LLMs misdiagnosing cardiovascular status when given fitness tracker data
- g947o 9mo agoI have www.msn.com added to the blocklist of my router's adblock settings so that I can stop seeing nonsense in Skype/Windows. It worked for a while. Of course, the real solution is to stop using Microsoft products, which I did.
- Aachen 9mo agoFrankly the main thing I take issue with is dumping these garbage titles onto HN. They can do what they want on their site
- eleveriven 9mo agoRight now this looks less like "AI for healthcare" and more like a very polished way to scare (or falsely reassure) people
- cthalupa 9mo agoI'll preface this with I generally trust doctors. I think on the whole they are well positioned to provide massive benefit to their patients. I will also preface this with saying I do not think any LLM is better than the average doctor and that you are far better served going to your doctor than asking ChatGPT what your health is like on any factor. But I'll also say that the quality of doctors varies massively, and that a good amount of doctors learn what they learn in school and do not keep up with the latest advances in research, particularly those that have broad spectrums such as GPs. LLMs that search scientific literature, etc., might point you in the direction of this research that the doctors are not aware of. Or hallucinate you into having some random disease that impacts 3 out of every million people and send you down a rabbithole for months. Unfortunately, it's difficult to resolve this without extremely good insurance or money to burn. The depth you get and the level of information that a good preventative care cardiologist has is just miles ahead of where your average family medicine practitioner is at. Statins are an excellent example - new prescriptions are for atorvastatin are still insanely high despite it being a fairly poor choice in comparison to rosuvastatin or pitavastatin for a good chunk of the people on it. They often are behind on the latest recommendations from the NLA and AHA, etc. There's a world where LLMs or similar can empower everyday people to talk to their doctor about their options and where they stand on health, where they don't have to hope their doc is familiar with where the science has shifted over the past 5-10 years, or cough up the money for someone who specializes in it. But that's not the world of today. In the mean time, I do think people should be comfortable being their own advocates with their doctors. I'm lucky enough that my primary care doc is open to reading the studies I send over to him on things and work with me. Or at least patient enough to humor me. But it's let me get on medications that treat my symptoms without side effects and improved my quality of life (and hopefully life/healthspan). There's also been things I've misinterpreted - I don't pick a fight with him if we come to opposite conclusions. He's shown good faith in agreeing with me where it makes sense to me, and pushed back where it hasn't, and I acknowledge he's the expert.
- biophysboy 9mo agoI think the fairest test is: what is the best and fastest way to reduce medical uncertainty? For rare ailments with a single cause and exclusive symptoms, that can be accurately described with simple language (no medical jargon), its possible that an LLM is better than a doctor. For more ambiguous situations where you need actual tests, I am skeptical of using LLMs.
- francisofascii 9mo ago> There were big swings in my resting heart rate whenever I got a new Apple Watch, suggesting the devices may not have been tracking the same way. First of all, wrist based HR measurements are not reliable. If you feed ChatGPT a ton of HR data that is just plain wrong, expect a bad result. Everyone who wants to track HR reliably should invest in a chest strap. The VO2 Max calculation is heavily based on your pace at a given heart rate. It makes some generalizations on on your running biomechanics. For example, if your "real" lab tested VO2 max stays constant, but you improve your biomechanics / running efficiency, you can run faster at the same effort, and your Apple watch will increase your VO2 Max number.
- AlanYx 9mo agoIn this case the article's guess is probably accurate. Apple did change how they measure RHR in WatchOS 11.2. If the author was using an Apple Watch that doesn't support 11.2 and then switched to one that does, a swing was very likely.
- zombot 9mo agoGiving your health data to an AI is sick. Unfortunately no doctor can cure you of that.
- hasbot 9mo agoHmm, sure it's maybe wrong now, but in several years, it could be correct. So maybe I should wear a device now so when it does become correct and I'm even older, AI might be useful. I'm definitely not going with Apple. Are there any minimally obtrusive trackers that provide downloadable data?
- spicyusername 9mo agoSo we're feeding bad data into a system known for making answers up and expecting... what exactly, lol
- zhisme 9mo agoCheck out iatrogenesis. There's no need to rely on apple watch data to become some drug addicted guy curing never existed diseases. That's not the metric you want to define whether you need meds and medical help at all.
- blef 9mo agoThe title would have been event better if: "I had ChatGPT analyze a decade of my Apple Watch data, then it called my doctor"
- ge96 9mo agoI think I found the proper sleep amount for me (sleep deprived). It has me feeling agitated/motivated. It's around 5 hours. What I need is something like Apple Watch to detect when I've actually fallen asleep then set the alarm for that long.
- jablongo 9mo agoThere needs to be more documentation about what info was provided to the LLM and in which format before we decide that LLMs are necessarily bad at this. That said, you would expect the offering from a $500bn company to be more robust and better tested than this, assuming this is reported accurately.
- uriegas 9mo agoThere are some research projects out there that use LLMs for health diagnostics. Here's one: https://cs.stanford.edu/people/jure/pubs/med-pmlr23.pdf https://cs.stanford.edu/people/jure/pubs/med-pmlr23.pdf They usually require more data It is not a great idea to diagnose anything with so few information. But in general I am optimistic of the use of LLMs on health.
- 6stringmerc 9mo agoIn my view the people, no matter walk of life or education level or societal class, who ask “AI” systems mental or physical health questions are modern day incarnations of customers who went to palm readers, tarot card sessions, or used to rely on televangelists for hope and meaning. I strongly dislike the author conflating HIPAA with PHI but this is a losing battle for me. And clearly editors don’t spot it, neither do AI systems - where is Clippy?! It simply serves as an indicator the author is a pretty ignorant medical consumer in the US, and this case study is stunning. Some people really should not be allowed to engage with magic.
- tiffanyh 9mo agoWhat's the feedback loop here for ChatGPT? For it to get better, it needs to know outcomes of its diagnosis. Are people just typing back to ChatGPT saying "you're wrong / you're right"?
- brewcejener 9mo ago[dead]
- rurban 9mo agoI use the free Huawei Health for like 2 years, and it was pretty good so far. The sensors suck of course, but better than nothing. I had a special watch to test my high blood pressure, but even this never matched my special pressure device.
- danielmarkbruce 9mo agovo2 max is one of the strongest predictors of all cause mortality. It's been reproduced across several large scale studies. I'm on the side of ChatGPT on this one. I'd guess the writer of this article is leaving something out. A family member recently passed away from a rare, clinically diagnosed disease. ChatGPT knew what it was a couple months before the relevant specialists diagnosed it.