3 ms·
Medprompt+: GPT-4 vs. Gemini Ultra
- verdverm 3y agoMisleading title, the first sentence of the Readme is... "promptbase is an evolving collection of resources, best practices, and example scripts for eliciting the best performance from foundation models like GPT-4" There happens to be one table in the Readme that compares the two foundational models on several benchmarks, but there is much more interesting content generally The essence of Medpromt+ is summarized (by MS) as In our Medprompt study, we focused on medical challenge problems, but found that the prompt strategy could have more general-purpose application and examined its performance on several out-of-domain benchmarks—despite the roots of the work on medical challenges. Today, we report that steering GPT-4 with a modified version of Medprompt achieves the highest score ever achieved on the complete MMLU.
- Racing0461 3y agoIs it possible for me to use these better prompts for my own use cases?
- verdverm 3y agoyes, I haven't seen anything really novel yet. There are not prompts you can simple copy out and use yourself, that's not what this is about. It's about Chain of Thought (CoT) and prompt engineering strategies. Right now, it's most showing how different CoT strategies perform with different models on different datasets. What's nice about it is they give you the code to reproduce their work. They also intend to keep adding more of their experiments and results, so a good repo to bookmark and check from time to time
- Racing0461 3y agoah ok. i thought it was something i can just paste into chatgpt. here's a 200$ tip for responding.
- verdverm 3y agoPrompting is not so simple, you'll find you need to update them as OpenAI updates the ChatGPT training and cutoff date Right now, there are a number of people complaining that ChatGPT has become "lazy", not filling in all details and using things like "etc..." or "you can fill in the rest"
- karmasimida 3y agoWhere did that HumanEval performance (zero-shot) gets reported? Did someone has the original paper? Would like to see it.
- sgt101 3y agoHow is this different from the Tree of Thought dumpster fire? It looks like an awful lot of fishing for a good answer is going on.
- choilive 3y agoInteresting, but correct me if I'm wrong... but does this technique only improve the performance on multiple choice questions where the correct answer is one of those choices? Unfortunately I don't see that as being very useful since I think a lot of use cases involve questions to the AI where you don't know the answer in advance or else why would you ask it except to test the AI's capabilities.