I think this is a key point - the training set is very important, because biases, over-curation, or wrong contexts will mean the model may perform very poorly f

I think this is a key point - the training set is very important, because biases, over-curation, or wrong contexts will mean the model may perform very poorly for particular scenarios or demographics. I can't find the reference now of a radiology AI system which had a good diagnosis rate of finding a pneumothorax on a chest x ray (air in the lining of the lung). This can be quite a serious condition, but is easy to miss. Turns out that the training set had a lot of 'treated' pneumothorax. The outcome was correct - they did indeed have a pneumothorax, but they also had a chest drain in, which was helping the prediction. Similar to asking what the demographic of training set is, is what the recorded outcome was. How was the diagnosis made. There is often no 'gold standard' of diagnosis, and some are made with varying degrees of confidence. Even a post-mortem can't find everything...