3 ms·
> “ Question: Write a detailed radiology note based on the chest X-ray. Gold Answer: AP upright and lateral views of the chest were provided. Left chest wall p
by davidhyde 2y ago
> “ Question: Write a detailed radiology note based on the chest X-ray.
Gold Answer: AP upright and lateral views of the chest were provided. Left chest wall pacer pack is again seen with leads extending into the right heart. ”
The bit about a “wall pacer pack is again seen…” leads me to believe this was based on another doctors note about a similar looking X-ray which was probably paired with other information like another scan at the time. That would be problematic imo.
- BriggyDwiggs42 2y agoProblematic for the functionality? If it works well enough, I’m pretty fine with them stealing data to create a useful medical tool.
- davidhyde 2y agoIt’s problematic because the LLM is describing another person’s scan and not the one presented to it. It should at least present the other scan as workings and the percentage difference between the two. Finding a similar looking scan is very useful no doubt but if the result is hallucinated that it is less so. Dangerous even. There is no confidence percentage and there should be.
- planb 2y agoThe Gold Answer is not the output of the model but the expected answer in the benchmark. Probably the benachmark contains multiple consecutive images of the same patient.
- davidhyde 2y agoThanks, I probably shouldn’t be commenting on an LLM article if I don’t even know what a gold answer is! Oh well.