5 ms·
Show HN: Medical LLM API on par with Google Med-PaLM 2. 92% USMLE accuracy
Hello HN!
I’d like to share a medical question-answering API which has state-of-the-art performance on the USMLE self-assessment exam. You can try out the MediSearch API in 10 seconds in this Colab [https://tinyurl.com/medisearch-colab https://tinyurl.com/medisearch-colab], or test the search engine live at MediSearch [https://medisearch.io https://medisearch.io]. See our API page [https://medisearch.io/api https://medisearch.io/api] to apply for access. Some technical highlights:
1. State-of-the-art accuracy of 92% on the USMLE self-assessment exam [https://tinyurl.com/medisearch-eval https://tinyurl.com/medisearch-eval].
2. Ranked 2nd on MedQA benchmark, closely tailing Google's Med-PaLM 2 (within ~2%).
3. The API provides article references as an output.
We’d love to hear feedback from this community! We’ve seen people use the MediSearch API to implement a chat bot or search field within their app.
Please note that MediSearch does not have a 100% accuracy. Therefore, a human professional should always make a final decision on the information suggested by MediSearch.
You can use the MediSearch API similarly to the OpenAI API [https://pypi.org/project/medisearch-client/ https://pypi.org/project/medisearch-client/]. If you’d like access to the MediSearch API, please email michal@medisearch.io or visit our API page [https://medisearch.io/api https://medisearch.io/api]. Please specify your use case in the application.
- mackopes 3y agoHaving an API with medical information is amazing! Just out of curiosity, is a React client or even a javascript library on your roadmap?
- mpmisko 3y agoHey, a React client is definitely on our roadmap! The demo (https://tinyurl.com/medisearch-docs https://tinyurl.com/medisearch-docs) in our docs is written in JS, so it should be useful out of the gate! A proper client is coming in the next few weeks.
- gardenfelder 3y agoI asked it a simple question: "what is known about PSC?" and it returned two citations. Interesting. Right now, nothing is known about the business model behind the product. What are the plans for it?
- mpmisko 3y agoThanks for giving it a go! We will charge for the API and we may have a pro subscription with some nice features :).
- hzoli42 3y agoWow, really cool to see citation support! What are the different sources that are cited?
- mpmisko 3y agoWe primarily cite medical scientific papers and health guidelines. We occasionally cite health blogs (such as healthline.com), but this is mostly for simple health questions (e.g., "tell me some nutrition tips").
- eoravkin 3y agoWe'd be interested to hear examples or ideas of how this could be useful for you!
- brandonb 3y agoAwesome! Can you talk a bit about how you trained or fine-tuned MediSearch's LLM?
- eoravkin 3y agoThanks! Glad that you like it:) We haven't trained or fine-tuned on neither of USMLE or MedQA. We use several LLMs in our system, and honestly I don't think there is any single thing that I can point to which is different to how other people do it. I guess it's a combination of many small improvements. Sorry for the vague answer. We are thinking about publishing a report at some point.
- dredev 3y agoThis is amazing! It almost never hallucinates. How did you guys accomplish that?
- eoravkin 3y agoThanks a lot! We put a lot of energy into solving hallucination. The tldr is that we have a eval set to test hallucination and tweak stuff to optimize for performance on this eval set.
- zopper 3y agoThis looks really cool. Does your API also support follow up questions and a longer conversations?
- mpmisko 3y agoGlad that you like it! We do support follow-up questions. Implementing a chat-style interaction should be very straightforward :) (see our Python client for examples https://pypi.org/project/medisearch-client/ https://pypi.org/project/medisearch-client/).