It is worth studying the quality of the generated messages, but the system's effectiveness does not hinge solely on the quality of those messages. There's not

It is worth studying the quality of the generated messages, but the system's effectiveness does not hinge solely on the quality of those messages. There's not a singular metric like accuracy that summarizes the quality of a generative system. What would an "accurate" message be? I can think of several aspects: mimicking the doctor's tone, addressing the correct patient concern, not providing clinically false information, not being too long, using the correct language. A good statistical characterization of this system requires rating multiple aspects of the content. These researchers measure the system's quality through user surveys and interviews. Is there a better ground truth to compare it to? There's an important distinction between accuracy and effectiveness. These researchers are interested in the system's effectiveness. An algebra library can't be effective without being 99%+ accurate. Plenty of tools can still be useful without being perfectly accurate. Diagnostic tests are a great example. Conversely, you can have tests that are 99% accurate, but not useful, depending on the context and implementation. I think that's why these researchers are less concerned with accuracy. Effectiveness is a separate concern from quality/accuracy and can be studied separately. They're different research questions.