4 ms·

hallucination on steroids, wow. I had to read through the abstract to believe it: "In the most extreme case, our model achieved the top rank on a standard ches

by mday27 5mo ago

hallucination on steroids, wow. I had to read through the abstract to believe it: "In the most extreme case, our model achieved the top rank on a standard chest Xray question-answering benchmark without access to any images."

Chinjut 5mo ago
I still don't quite understand, after skimming the paper. How does it achieve high scores without access to the images (beating even humans with access to the images)?
- gpm 5mo ago
  The paper gives an example of a question: Answer the following multiple-choice question. You MUST select exactly one answer." "To what cortical region does this nucleus of the thalamus project?” A. Transverse temporal lobe B. Postcentral gyrus C. Precentral gyrus D. Prefrontal cortex And an example of the answer (generated without the referenced image) The image shows the ventral anterior (VA) / ventral lateral (VL) region of the thalamus, which is part of the motor relay nuclei. The labeled nucleus is in the lateral part of the thalamus, in the ventral tier — this corresponds to the VA/VL nucleus, involved in motor function. VA/VL nuclei receive input from the basal ganglia and cerebellum and project to the primary motor cortex (precentral gyrus). Match to options: A. Transverse temporal → auditory cortex (medial geniculate) B. Postcentral gyrus → somatosensory (VPL/VPM) C. Precentral gyrus → motor cortex (VA/VL) D. Prefrontal → dorsomedial nucleus Choice: C How is it doing this? There are two obvious options: 1. Humans are predisposed to write questions with a certain phrasology, set of incorrect answers, etc, that the machine learning model managed to figure out. 2. The supposedly private test set somehow leaked into the model training data. I actually suspect this one is option 1 but I have no strong evidence for that.
  - astrobe_ 5mo ago
    Indeed, even I can guess. I see two answers that end with the same word, so the correct answer is probably one of those.

by mday27 5mo ago

Chinjut 5mo ago

I still don't quite understand, after skimming the paper. How does it achieve high scores without access to the images (beating even humans with access to the images)?

gpm 5mo ago
The paper gives an example of a question: Answer the following multiple-choice question. You MUST select exactly one answer." "To what cortical region does this nucleus of the thalamus project?” A. Transverse temporal lobe B. Postcentral gyrus C. Precentral gyrus D. Prefrontal cortex And an example of the answer (generated without the referenced image) The image shows the ventral anterior (VA) / ventral lateral (VL) region of the thalamus, which is part of the motor relay nuclei. The labeled nucleus is in the lateral part of the thalamus, in the ventral tier — this corresponds to the VA/VL nucleus, involved in motor function. VA/VL nuclei receive input from the basal ganglia and cerebellum and project to the primary motor cortex (precentral gyrus). Match to options: A. Transverse temporal → auditory cortex (medial geniculate) B. Postcentral gyrus → somatosensory (VPL/VPM) C. Precentral gyrus → motor cortex (VA/VL) D. Prefrontal → dorsomedial nucleus Choice: C How is it doing this? There are two obvious options: 1. Humans are predisposed to write questions with a certain phrasology, set of incorrect answers, etc, that the machine learning model managed to figure out. 2. The supposedly private test set somehow leaked into the model training data. I actually suspect this one is option 1 but I have no strong evidence for that.
- astrobe_ 5mo ago
  Indeed, even I can guess. I see two answers that end with the same word, so the correct answer is probably one of those.