Either way, this is a good one for debate given the use is assumed to require careful attention to outputs. Giving someone with a mental health condition advice

Either way, this is a good one for debate given the use is assumed to require careful attention to outputs. Giving someone with a mental health condition advice is fraught with peril, even in a clinical setting conducted by humans with vast experience. The transcripts of therapy sessions would be quite helpful for improved training of the model, given they contain logic that may not be present in the limited dataset provided. It would be a hope that these detailed interactions would provide improvement into the model's problem solving capacity for helping those with mental illness. As an example, for certain conditions the model may use a more cautious approach to investigation of the source of trauma. That's not to say therapy session transcripts should be used for prompt tuning after training, which exposes the data directly to the inference pipeline. However, we do know fine tuning the prompts with data is certainly useful for grounding, at the very least. I'm making the argument that using sensitive data in training is exactly what makes the model better, not filtered data that lacks the wide variety of "expertise" and "experience" that is contained in more sensitive datasets. If this were done, we can all reasonably assume that "sensitive data" will make its way into the tensors of the model, but the question is whether or not that information is more valuable for everyone's general use compared to the risk of outputting something that diminishes the value for an individual. I'm not proposing this is a good idea to do, but thinking about it is certainly worthwhile.