3 ms·

Really? I have been doing research on language models in medical diagnostics even before GPT-2, and found that when trained and applied in certain ways, languag

Really? I have been doing research on language models in medical diagnostics even before GPT-2, and found that when trained and applied in certain ways, language models (even much smaller than GPT-3!) are very good at diagnosis predictions, they can compete with much more complex symptom checkers at that. Proof: Link to my paper (written back in 2019) and a bit less technical article. http://www.dialog-21.ru/media/4632/tarasovdplusetal-069.pdf http://www.dialog-21.ru/media/4632/tarasovdplusetal-069.pdf https://www.linkedin.com/pulse/language-models-multi-purpose-medical-ai-systems-denis-tarasov https://www.linkedin.com/pulse/language-models-multi-purpose... I applied for GPT-3 access on the next day since the application form was available, described my research and experience in detail, but there was no reply. Now, they gave access to these people at nabla, and they just asked a bunch of stupid questions using top-k random sampling to generate answers and claimed that this debunks something. This study debunks nothing and proves nothing, it is stupid and only done to get some hype from GPT-3 popularity. Ok, I am sorry for being rude, but I am really upset because I spent years working on this problem using whatever computational resources I could get and obtained some interesting results, and based on these I think that GPT-3 should be capable to do amazing things for diagnostics when used properly. Why won't OpenAI give access to a researcher who wants to do some serious but a bit mundane work, but gives it to people who use it to create hype?

2-tpg 6y ago
I used GPT-2 to create a health website. One sentence was enough to get a full page of authoritatively sounding lists of symptoms and treatments. Very diverse, unlike all other sites, because the articles it generated only looked and sounded like a health encyclopedia. Of course it is going to spit back decent diagnosis, when it is in the training data, but what do you trust? An expert system that logically and interpretable explains its predictions, linking the original source. Or a language model that uses a temperature to stay on track, and randomizes its output on every new run? Generating data with a possible high impact on lives sounds like a recipe for disaster and frankly, irresponsible. And Google would have to really solve it, to detect false or questionable information, when its not possible to rely on spam signals (like when a legit site is transferred to a malicious spammer). Aside, I bet LeCun would be more favorable of GPT-3 had it been a deep CNN and they had adopted his self-supervised learning paradigm :).
- nombinoms 6y ago
  It is self-supervised learning. Specifically, a masking auto encoder.

2-tpg 6y ago

I used GPT-2 to create a health website. One sentence was enough to get a full page of authoritatively sounding lists of symptoms and treatments. Very diverse, unlike all other sites, because the articles it generated only looked and sounded like a health encyclopedia. Of course it is going to spit back decent diagnosis, when it is in the training data, but what do you trust? An expert system that logically and interpretable explains its predictions, linking the original source. Or a language model that uses a temperature to stay on track, and randomizes its output on every new run? Generating data with a possible high impact on lives sounds like a recipe for disaster and frankly, irresponsible. And Google would have to really solve it, to detect false or questionable information, when its not possible to rely on spam signals (like when a legit site is transferred to a malicious spammer). Aside, I bet LeCun would be more favorable of GPT-3 had it been a deep CNN and they had adopted his self-supervised learning paradigm :).

nombinoms 6y ago
It is self-supervised learning. Specifically, a masking auto encoder.