A few years ago the NHS gave a load of confidential patient data to an insurance company to train their models. They claimed that it wasn't a problem because th

A few years ago the NHS gave a load of confidential patient data to an insurance company to train their models. They claimed that it wasn't a problem because the data was deleted after training. However, nobody knows (publicly) what kind of model it was. If they overfitted the data then when you ask for an insurance quote based on say, date of birth and postcode, the quote will be based on your actual medical details rather than on broader statistics. The result is no different from them having explicit access to your medical records before giving a quote, which is not permitted. It is easy to build an ML system which for most queries provides the same answer as having access to the original data.