6 ms·
One thing that jumps out is that the paper fails to provide a proper direct comparison with Med-PaLM, only mentioning the latter in passing almost as an afterth
by aix1 4y ago
One thing that jumps out is that the paper fails to provide a proper direct comparison with Med-PaLM, only mentioning the latter in passing almost as an afterthought. Looking at arXiv timestamps, the Med-PaLM preprint has been out of three months (given the rate at which this field moves, an eternity). I understand Med-PaLM the current state of the art and would be a natural point of comparison.
https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138
- ShamelessC 4y ago> Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). What’s your contention exactly? Synopsis mentions it, does the paper not follow up on it?
- aix1 4y agoMed-PaLM has been tested on a number of public benchmarks, and the results published. A useful comparison would have been to test GPT on the same set of benchmarks. Yet the paper we're discussing offers just one datapoint by way of a numerical comparison, literally in a footnote. Basically, the comparison needs to be a lot more comprehensive to be useful.
- FlyingLawnmower 4y agoLate to the thread here, but the paper announcing Med-PaLM (https://arxiv.org/abs/2212.13138 https://arxiv.org/abs/2212.13138) does not report many benchmark results on Med-PaLM and is instead mostly about Flan-PaLM 540B (which is compared against in this paper). I am curious if any other Med-PaLM benchmarks have been published but I don't believe it is currently possible to do any further comparison against Med-PaLM given that the model is not public and no other open benchmark results are reported in the original Med-PaLM paper.