3 ms·

Some of my former co-workers founded a start-up a few years ago (before the ChatGPT age) focussing on AI tools for the medical sector. By far their biggest chal

by kleiba 3y ago

Some of my former co-workers founded a start-up a few years ago (before the ChatGPT age) focussing on AI tools for the medical sector. By far their biggest challenge wasn't the tech but the hoops they had to jump through to get even their smallest tools certified so that they could be used by doctors at all. Unbelievable amounts of red tape and time investment on the side of that startup. I find it disgusting to hear that ChatGPT and the likes are now regularly used by medical professionals where, obviously, these LLM tools have never gone through the same process. For two reasons - one because of the obvious medical risks involved (hallucinating LLMs) but also because of the gross commercial advantage the companies behind these LLMs get without even as much as trying to get certified. I've lost touch with my former co-workers but I would be surprised if they hadn't looked into LLMs to be included in their products. My guess is that they're still somewhere in the administrative hell of trying to get their latest tools certified while meanwhile, doctors are happily using ChatGPT and GenAI. Must be frustrating.

harvey9 3y ago
If doctors consult chatgpt (unregulated, uncertified) that's on them, not the platform. I hope your old team keep working on high quality stuff but I can understand if they walk away. I have seen some chatbot type products aimed at the admin side of healthcare, with the salesmen telling me about the 'guardrails'. Given some guy recently got the bot at his utility supplier to start criticising the company and using foul language then I do not think even this less critical work is ready for general use.
- kleiba 3y ago
  Well, I think it's worse than you may think: the startup is not even allowed to market their products as medical tools unless they're certified. ChatGPT is used today by doctors without being certificied. How is that fair?
  - harvey9 3y ago
    I don't think it is fair and I do know the difference between a regulated product and an ad hoc (mis)use of ChatGPT. I just do not think you can blame OpenAI for doctors (or other professionals) ignoring the warning they are presented with. On the other hand I do think professional bodies should be inclined to strike off members who do this kind of thing.
  - Kerb_ 3y ago
    Doctors use Google, Google was marketed as a search engine and not a medical tool. Seems like if ChatGPT is advertised as a language model and not a medical model, it's completely fair. Irresponsible for doctors to rely on, maybe, but I don't see why a non-medical tool needs to be medically certified just because doctors choose to use it
    - kleiba 3y ago
      Then why do we need certification at all? Everyone just brand your tool as non-medical, not matter whether it is intended for medical use or not, and Bob's your uncle!
      - Kerb_ 3y ago
        Because doctors are liable for improperly using non-medical means to diagnose and cure illnesses, even if they might theoretically help. If I develop a new design of pliers for my skateboard that end up working really well for performing throat surgery, you're free to go and make a similar product and get that certified. I don't see why mine needs to be certified. I didn't make it for medical use. GPT explicitly tells you not to use it for diagnosis purposes, how many doctors need to use something before it's forced to get certified despite its intended use?
btbuildem 3y ago
It quickly became apparent that LLM are "expert assistants" (as in, assistants for experts). I use them at work to take shortcuts - eg, I won't look up the syntax for some function call, I'll ask GPT instead. I'm able to evaluate whether it's correct or not (and often some small thing will be off) but it's still a quicker way to get a good result than manually synthesizing it from two or three different references. I imagine it's similar with doctors -- an expert can get a quick 80% answer, sound it against their judgement, and refine.

by kleiba 3y ago

harvey9 3y ago

If doctors consult chatgpt (unregulated, uncertified) that's on them, not the platform. I hope your old team keep working on high quality stuff but I can understand if they walk away. I have seen some chatbot type products aimed at the admin side of healthcare, with the salesmen telling me about the 'guardrails'. Given some guy recently got the bot at his utility supplier to start criticising the company and using foul language then I do not think even this less critical work is ready for general use.

kleiba 3y ago
Well, I think it's worse than you may think: the startup is not even allowed to market their products as medical tools unless they're certified. ChatGPT is used today by doctors without being certificied. How is that fair?
- harvey9 3y ago
  I don't think it is fair and I do know the difference between a regulated product and an ad hoc (mis)use of ChatGPT. I just do not think you can blame OpenAI for doctors (or other professionals) ignoring the warning they are presented with. On the other hand I do think professional bodies should be inclined to strike off members who do this kind of thing.
- Kerb_ 3y ago
  Doctors use Google, Google was marketed as a search engine and not a medical tool. Seems like if ChatGPT is advertised as a language model and not a medical model, it's completely fair. Irresponsible for doctors to rely on, maybe, but I don't see why a non-medical tool needs to be medically certified just because doctors choose to use it
  - kleiba 3y ago
    Then why do we need certification at all? Everyone just brand your tool as non-medical, not matter whether it is intended for medical use or not, and Bob's your uncle!
    - Kerb_ 3y ago
      Because doctors are liable for improperly using non-medical means to diagnose and cure illnesses, even if they might theoretically help. If I develop a new design of pliers for my skateboard that end up working really well for performing throat surgery, you're free to go and make a similar product and get that certified. I don't see why mine needs to be certified. I didn't make it for medical use. GPT explicitly tells you not to use it for diagnosis purposes, how many doctors need to use something before it's forced to get certified despite its intended use?

btbuildem 3y ago

It quickly became apparent that LLM are "expert assistants" (as in, assistants for experts). I use them at work to take shortcuts - eg, I won't look up the syntax for some function call, I'll ask GPT instead. I'm able to evaluate whether it's correct or not (and often some small thing will be off) but it's still a quicker way to get a good result than manually synthesizing it from two or three different references. I imagine it's similar with doctors -- an expert can get a quick 80% answer, sound it against their judgement, and refine.