I think people like the hallucinations because they seem like the right answer. Imagine I make an AI assistant. You ask it, "how do I catch the bus into town?

I think people like the hallucinations because they seem like the right answer. Imagine I make an AI assistant. You ask it, "how do I catch the bus into town?" If it replies, "run around in your backyard in a circle, then clap 3 times", (i.e. just use a random number generator) you'll think that's junk and will hate the system. But if it says "go to the corner of Main and Pine at 6:53 AM", then you'll probably do that. No matter that there is no bus route on Main or Pine. It sounds right, so you'll want to follow up. I imagine the same applies to searching for novel molecules. "This looks kind of right, I never would have thought of that", so you get a grant to synthesize it, do some clinical trials, and find that it doesn't work at all. But hey, you had something to do for a couple years, and that outcome happened in the non-AI process all the time too. By the time there is enough data that points strongly at the AI suggestions being as good as random chance, the AI gold rush will be over and cashed out. Or, the hallucinations will turn out to be really good and we'll cure cancer or whatever. What matters is that whoever made the model makes a lot of money either way.