If this is remotely like my experience with ChatGPT, doesn't that run the risk of hallucinating? Versus pattern matching just telling you "This X-ray shows a 75

If this is remotely like my experience with ChatGPT, doesn't that run the risk of hallucinating? Versus pattern matching just telling you "This X-ray shows a 75% chance this person has condition X". Can you at least interact with it and have it explain to you its reasoning like "so my diagnostic is based on seeing a lump in the lower right hand corner"? That would be an innovation indeed.