A way I've been thinking about this today is: We can't distinguish between a truly novel response from an LLM or a hallucination. We can get some of the way t

A way I've been thinking about this today is: We can't distinguish between a truly novel response from an LLM or a hallucination. We can get some of the way there, such as if we know what the outcome to a problem should look like, and are seeking a better function to achieve that outcome. Certainly at small scales and in environments where there are minimal consequences for failure, this could work. But this breaks down as things get more complicated. We won't be able to test the effectiveness of 100 million potential solutions to eradicating brain tumors at once. Even if we somehow arrive at guaranteeing that every unforeseen consequence is also accounted for in our exercise in specifying the goals and constraints of the problem. We just simply don't have the logistics to run 100 million clinical trials where we also know how to account for countless confounding effects (let alone consent!)