3 ms·
None of these were binary decisions, but classifying one of around 10-20 conditions or rating cases on a 1-5 scale. In all cases the models trained on a lot of
by m_ke 2y ago
None of these were binary decisions, but classifying one of around 10-20 conditions or rating cases on a 1-5 scale.
In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.
- Uehreka 2y agoI’m guessing these are also specially trained image classifiers and not LLMs, so people’s intuitions about how LLMs work/fail may not apply.
- m_ke 2y agoIt’s the same softmax classifier
- snowwrestler 2y agoWait if experts only agreed 60% on diagnoses, what is the reliable basis for judging LLM accuracy? If experts struggle to agree on the input, how are they confidently ranking the output?
- petra 2y agoYou can look at fully diagnosed cases(via surgery for example) and their previous scans.
- throwup238 2y agoNot the OP but the data isn’t randomly selected, it’s usually picked out of a dataset with known clinical outcomes. So for example if it’s a set of images of lungs with potential tumors, the cases come with biopsies which determined whether it was cancerous or just something like scar tissue.
- Eisenstein 2y agoPerhaps they were from cases that had a confirmed diagnosis.