2 ms·
This is incredibly misleading. The actual study: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309?guestAccessKey=6d6e7fbf-54c1-49fc-8f5
by ceejayoz 4y ago
This is incredibly misleading. The actual study: https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309?guestAccessKey=6d6e7fbf-54c1-49fc-8f5e-ae7ad3e02231 https://jamanetwork.com/journals/jamainternalmedicine/fullar...
Buried deep in the limitations section:
> evaluators did not assess the chatbot responses for accuracy or fabricated information
Repeat:
> EVALUATORS DID NOT ASSESS THE CHATBOT RESPONSES FOR ACCURACY OR FABRICATED INFORMATION
- sebzim4500 4y agoBut they did assess "the quality of information provided". I don't understand what that means if not accuracy.
- ceejayoz 4y agoAs far as I can determine, they basically did a vibe check. “Sounds good to me” sort of thing.
- danShumway 4y agoThis should be rated higher, possibly even over the reveal that this study is comparing to Reddit answers. https://xkcd.com/937/ https://xkcd.com/937/ comes to mind. It's not implausible at all to me that ChatGPT could outperform Reddit in detail/manners for health advice (and honestly, even for actual doctors I've heard some horror stories about bedside manner and refusing to actually believe/consider symptoms), but if the study isn't actually checking that, if they're just checking if the chatbot was more polite/empathetic... that's a huge qualification that should be up-front and center.