3 ms·
I am highly unimpressed by this piece. It reads as if its whole purpose is to grab headlines rather than conduct serious scientific inquiry into the current st
by aix1 4y ago
I am highly unimpressed by this piece. It reads as if its whole purpose is to grab headlines rather than conduct serious scientific inquiry into the current state -- and limitations -- of these AI methods.
1. Which version of GPT did the author use? There's a huge difference. (The article says "the current version".)
2. How did he choose the subject cohort? (The author doesn't seem to even know how many subjects there were; the article says "35 to 40 patients"... I really do hope he's gone through an appropriate ethics review before feeding his patients' data into a third-party non-HIPAA system.)
3. There no evidence of him trying to get the best out of the model (eg through prompt engineering).
4. He assumes that his own diagnoses are 100% correct.
5. There is no control group (other doctors diagnosing the same patients).
and so on
- lapcat 4y agoIt's a blog post, not a journal article. You're criticizing it as if it were a funded, peer-reviewed experiment, which seems unfair to me. The author is just one ER doctor.
- Pigalowda 4y agoAn IRB for deidentified HPI? Got it…
- aix1 4y agoExactly. I've been through IRB reviews where the primary question was "Has the data been de-identified to a sufficient standard?" I think this level oversight would be very appropriate here, given how the author doesn't even seem to have a good handle on how many patient case histories he's given to the chatbot.