16 ms·

Google’s DeepMind made ‘inexcusable’ errors handling UK health data, says report

93 points by aaronyy 10y ago

cryptoz 10y ago
Note: The errors are by people and business dealings, not in decisions made by AI while analyzing health data.
koolba 10y ago
> The data-sharing agreement — which was signed in 2015 and has since been superseded by a new contract — allows DeepMind access to medical records from 1.6 million patients attending London hospitals run by the NHS Royal Free Trust. Although at the time Google presented the deal as primarily about finding patients at risk from a condition known as acute kidney injury or AKI, the actual terms of the agreement, revealed in April 2016 by a New Scientist investigation, were more broad. > The report notes that DeepMind was given access not only to relevant blood tests and diagnostics, but historical medical records dating back five years, including information on HIV diagnoses, drug overdoses, and abortions. The report also says the wording of the 2015 deal did not constrain the company from using AI analytical techniques on the data (something DeepMind disputes). What's the legal status and overall vibe of something like this in the UK? It's a bit different in the USA as we don't really have an NHS here with everybody's data and it'd be done directly with multiple insurers or medical providers. I'm guessing this would violate some type of patient privacy laws as well. Given the option I bet many people, including your humble commenter, would opt-out too. (or just not opt-in if we're lucky).
- throwaway729 10y ago
  https://www.mib.com/ https://www.mib.com/ > including your humble commenter, would opt-out too. (or just not opt-in if we're lucky). At this moment, do you know what organizations own a copy of your medical records -- in whole or part -- and what laws apply to each of those partial records? If not, then how can you possibly hope to opt-out?
  - koolba 10y ago
    > At this moment, do you know what organizations own a copy of your medical records -- in whole or part -- and what laws apply to each of those partial records? If not, then how can you possibly hope to opt-out? I don't and that's partly why I prefixed that line with, "Given the option ..."
    - throwaway729 10y ago
      I see, I thought you meant "give the option to opt-out". As in the option to opt-out is the missing thing, rather than knowing who you should even ask for an opt-out.
- deleted 10y ago
  [deleted]
  - deleted 10y ago
    [deleted]
- Silhouette 10y ago
  What's the legal status and overall vibe of something like this in the UK? The most basic issue is that health records about an identifiable individual are classified as "sensitive personal data" under the Data Protection Act, which is our primary privacy legislation. As such, there are several extra conditions that apply, in addition to all the ones covering all personal data, which constrain how "data controllers" and "data processors" are allowed to use that data. (The technical terms are defined in the Act itself.) A lot of the details discussed in the original report are relevant because in this case the "data subjects" (the patients) did not give their explicit consent. There are specific conditions required under the Act for processing sensitive personal data without such consent, and it's not clear whether this agreement met them, for reasons such as those discussed in the report we're talking about. If in fact the conditions were not met then a lot of people have probably broken the law, and both the organisations involved and their officers are potentially guilty of offences.
- DanBC 10y ago
  > It's a bit different in the USA as we don't really have an NHS here with everybody's data We don't really have that in the UK. This is one particular NHS Trust that runs 3 (I think) hospitals (and some other services) in London. https://www.royalfree.nhs.uk/ https://www.royalfree.nhs.uk/ So if I go to them to have my knee replaced, and I go to a different trust to have a lump-ectomy, the data Deepmind gets doesn't include (I think) the lumpectomy. As for your question about whether this kind of thing is popular: a while ago the NHS wanted to run something called Care.Data (care dot data) which was a big data project. A bunch of people complained about not being allowed to opt out, and so an opt out was added, and a bunch of people then opted out. Personally, I would have opted in if they'd given that option.
MistahKoala 10y ago
This has the whiff of what might kindly be called activist research, particularly given the track record of the author(s). There may be legitimate questions and concerns, but I'm inclined to trust the Wellcome representative's analysis of the situation.
joatmon-snoo 10y ago
Errors in transparency and oversight. Fix the title please. Still a very legitimate concern, especially in the age of privacy.
aub3bhat 10y ago
Frankly these post-docs are engaging in outright witchhunt against DeepMind driven by nothing but pure political agenda. The amount of data obtained is frankly tiny compared to that available regularly to researchers in USA. E.g. as part of my PhD research I have access to de-indentifed data on 40 Million patients spanning 5 years from several states. Not to forget programs like CMS Qualified Entity which provide access to identifiable data on ALL Medicare enrolees to companies. Such arrangements have been in place for decades. The only reason Deep Mind is being persecuted is since they are a juicy target, and fear of AI sells very well these days in academic circles. The report does not lists any specific cases where violations occured but rather makes broad hand waving claims about cabining or advertising. For those interested here is the orginal paper, instead of verge article. https://link.springer.com/article/10.1007%2Fs12553-017-0179-1 https://link.springer.com/article/10.1007%2Fs12553-017-0179-... If they really had substantive argument (that indicated real malice on part of Deep Mind) it would have been easily published in Lancet, BMJ, even New England Journal of Medicine. Instead the fact its published in some subject specific Journal should tell you about concreteness of their "findings".
- stuaxo 10y ago
  This sort of thing is important in a context where more and more NHS services are getting sold off to private entities. Arrangements that have been in the place for decades in the US are not necessarily relevant to the UK.
  - aub3bhat 10y ago
    "What sort of thing?" The report essentially says Google owns Deep Mind and Google has advertising business, ipso facto, some magical standrard dreamt up by the authors were not met. The reason for mentioning US agreements is that several nations, researchers and comapnies have developed procedures around sharing this data. And such sharing is not entirely unprecedented.
    - Silhouette 10y ago
      You're downplaying valid concerns. The ability to de-anonymise large data sets that we have today is unprecedented. The degree to which systems with access to that data are accessible remotely and potentially vulnerable to security problems is unprecedented. The degree to which powerful organisations like employers and insurers are attempting to profile potential employees and customers is unprecedented. However, neither major breaches involving huge amounts of potentially sensitive personal data nor scope creep in how data is used by large organisations once acquired are unprecedented. None of this is good for the individuals whose data is potentially at risk, and it's perfectly fair and reasonable to ask whether some of the most sensitive data there is about individuals is being properly handled by those entrusted with it given the implications of modern technology. There might be a lot of potential good in big data analysis for improving healthcare outcomes, but there is also a lot of potential harm if people stop, say, being confident in discussing potentially limiting conditions with their doctors, or reaching out for help with mental health issues because they're worried about confidentiality.
tomxor 10y ago
I don't want my privacy invaded as much as the next person, but arguing about broadness of medical history in this context is pretty stupid. The data needs to be broad if you are interested in finding out things you don't already know... that's why it's being fed into a machine learning algorithm in the first place - If you get selective then it's not going to be very useful - how do you limit history to what's relevant when you don't know whats relevant? There is however an interesting difference here between most data mining on the web which is trying to sell advertising. In this instance it should be fully anonymisable, only the doctor should be allowed to see that patient ID e76f57a is John Smith.
- 3JPLW 10y ago
  Note that there's a big difference between de-identifying and fully anonymizing data. In general, it's extremely hard to fully anonymize a dataset. De-identification gets you most of the way there, but its frequently possible for a dedicated attacker to re-identify users by combining the data with public datasets. For example, patient ID e76f57b may be a 38 year old woman in Smalltown. Her medical record states that she gave birth on 3-20-2017. Find all public birth announcements from that day in that town with a 38 year old mother. Once you have a small set of candidates, it's not hard to narrow down farther.
akamaka 10y ago
You can skip reading the article, as it does not list any "errors" that have happened. It merely questions whether the agreement under which the data is shared has adequate protections.
- 3JPLW 10y ago
  Indeed. The paper itself details seven "transgressions:" > 1) We do not know––and have no power to find out––what Google and DeepMind are really doing with NHS patient data, nor the extent of Royal Free’s meaningful control over what Google and DeepMind are doing; > 2) Any assurances about use of the dataset come from public relations statements, rather than independent oversight or legally binding documents; > 3) The amount of data transferred is far in excess of the requirements of those publicly stated needs, but not in excess of the information sharing agreement and broader memorandum of understanding governing the deal, both of which were kept private for many months; > 4) The data transfer was done without consulting relevant regulatory bodies, with only one superficial assessment of server security, combined with a post-hoc and inadequate privacy impact assessment; > 5) None of the millions of identified individuals in the dataset were either informed of the impending transfer to DeepMind, nor asked for their consent; > 6) The transfer relies on an argument that DeepMind is in a “direct care” relationship with each patient that has been admitted to Royal Free constituent hospitals, even though DeepMind is developing an app that will only conceivably be used in the treatment of one sixth of those individuals; and > 7) More than 12 months into the deal being made, no regulator had issued any comment or pushback. Quite a few of these strike me as rather absurd, but I don't know the regulatory environment in the UK.
  - Silhouette 10y ago
    Most of these issues potentially involve direct violations of both standard practices within our healthcare regulatory framework and explicit data protection laws. They aren't absurd at all. They're talking about doing an end-run around our most fundamental privacy safeguards, in relation to some of the most sensitive personal data that exists about any given individual.
  - throwaway729 10y ago
mikecb 10y ago
Interestingly, Deepmind's Ben Laurie has been working on certificate transparency-like tech to permit a verifiable audit log of access to data like this, precisely for this purpose: https://qz.com/929833/googles-goog-deepmind-is-using-blockchain-technology-to-handle-nhs-medical-data/ https://qz.com/929833/googles-goog-deepmind-is-using-blockch... If you want to take a look at the underlying technology, take a look at https://github.com/google/trillian https://github.com/google/trillian
KCFforecast 10y ago
Under Spain protection of personal data law, any file with personal information must be accessible for the person, and the person has the right to know, modify and delete that file, and to deny the access to that information for any purpose.
- desas 10y ago
  There must be qualifiers, you can't tell your bank to forget the money you owe it.
  - KCFforecast 10y ago
    You are right, more information here: http://uk.practicallaw.com/1-520-8264 http://uk.practicallaw.com/1-520-8264
biggio 10y ago
I don't care or why should I care? As long as there's progress in making NHS more efficient in treating me that's fine. If they need it I will personally go to their offices and give blood samples or whatever they need on a daily basis!
Gatsky 10y ago
"In July 2015, clinicians from British public hospitals within the Royal Free London NHS Foundation Trust approached Google DeepMind Technologies Limited, an artificial intelligence company with no experience in providing healthcare services, about developing software using patient data from the Trust." Actually, the institution which collects and stores the data handed it over without due process. The article keeps trying to blame DeepMind, I guess criticising the NHS is a little stale. I think this is in a journal because it is too long for an op-ed but not substantial enough for a long form article.
killjoywashere 10y ago
I would urge folks writing these deals to make sure to separate the rights assigned to 1) algorithms, 2) data, and 3) the models they generate. There is a clear joint interest in the models, and that seems to get missed in most of these articles.

93 points by aaronyy 10y ago

Note: The errors are by people and business dealings, not in decisions made by AI while analyzing health data.

> The data-sharing agreement — which was signed in 2015 and has since been superseded by a new contract — allows DeepMind access to medical records from 1.6 million patients attending London hospitals run by the NHS Royal Free Trust. Although at the time Google presented the deal as primarily about finding patients at risk from a condition known as acute kidney injury or AKI, the actual terms of the agreement, revealed in April 2016 by a New Scientist investigation, were more broad. > The report notes that DeepMind was given access not only to relevant blood tests and diagnostics, but historical medical records dating back five years, including information on HIV diagnoses, drug overdoses, and abortions. The report also says the wording of the 2015 deal did not constrain the company from using AI analytical techniques on the data (something DeepMind disputes). What's the legal status and overall vibe of something like this in the UK? It's a bit different in the USA as we don't really have an NHS here with everybody's data and it'd be done directly with multiple insurers or medical providers. I'm guessing this would violate some type of patient privacy laws as well. Given the option I bet many people, including your humble commenter, would opt-out too. (or just not opt-in if we're lucky).

throwaway729 10y ago
https://www.mib.com/ https://www.mib.com/ > including your humble commenter, would opt-out too. (or just not opt-in if we're lucky). At this moment, do you know what organizations own a copy of your medical records -- in whole or part -- and what laws apply to each of those partial records? If not, then how can you possibly hope to opt-out?
- koolba 10y ago
  > At this moment, do you know what organizations own a copy of your medical records -- in whole or part -- and what laws apply to each of those partial records? If not, then how can you possibly hope to opt-out? I don't and that's partly why I prefixed that line with, "Given the option ..."
  - throwaway729 10y ago
    I see, I thought you meant "give the option to opt-out". As in the option to opt-out is the missing thing, rather than knowing who you should even ask for an opt-out.
deleted 10y ago
[deleted]
- deleted 10y ago
  [deleted]
Silhouette 10y ago
What's the legal status and overall vibe of something like this in the UK? The most basic issue is that health records about an identifiable individual are classified as "sensitive personal data" under the Data Protection Act, which is our primary privacy legislation. As such, there are several extra conditions that apply, in addition to all the ones covering all personal data, which constrain how "data controllers" and "data processors" are allowed to use that data. (The technical terms are defined in the Act itself.) A lot of the details discussed in the original report are relevant because in this case the "data subjects" (the patients) did not give their explicit consent. There are specific conditions required under the Act for processing sensitive personal data without such consent, and it's not clear whether this agreement met them, for reasons such as those discussed in the report we're talking about. If in fact the conditions were not met then a lot of people have probably broken the law, and both the organisations involved and their officers are potentially guilty of offences.
DanBC 10y ago
> It's a bit different in the USA as we don't really have an NHS here with everybody's data We don't really have that in the UK. This is one particular NHS Trust that runs 3 (I think) hospitals (and some other services) in London. https://www.royalfree.nhs.uk/ https://www.royalfree.nhs.uk/ So if I go to them to have my knee replaced, and I go to a different trust to have a lump-ectomy, the data Deepmind gets doesn't include (I think) the lumpectomy. As for your question about whether this kind of thing is popular: a while ago the NHS wanted to run something called Care.Data (care dot data) which was a big data project. A bunch of people complained about not being allowed to opt out, and so an opt out was added, and a bunch of people then opted out. Personally, I would have opted in if they'd given that option.

MistahKoala 10y ago

This has the whiff of what might kindly be called activist research, particularly given the track record of the author(s). There may be legitimate questions and concerns, but I'm inclined to trust the Wellcome representative's analysis of the situation.

joatmon-snoo 10y ago

Errors in transparency and oversight. Fix the title please. Still a very legitimate concern, especially in the age of privacy.

aub3bhat 10y ago

Frankly these post-docs are engaging in outright witchhunt against DeepMind driven by nothing but pure political agenda. The amount of data obtained is frankly tiny compared to that available regularly to researchers in USA. E.g. as part of my PhD research I have access to de-indentifed data on 40 Million patients spanning 5 years from several states. Not to forget programs like CMS Qualified Entity which provide access to identifiable data on ALL Medicare enrolees to companies. Such arrangements have been in place for decades. The only reason Deep Mind is being persecuted is since they are a juicy target, and fear of AI sells very well these days in academic circles. The report does not lists any specific cases where violations occured but rather makes broad hand waving claims about cabining or advertising. For those interested here is the orginal paper, instead of verge article. https://link.springer.com/article/10.1007%2Fs12553-017-0179-1 https://link.springer.com/article/10.1007%2Fs12553-017-0179-... If they really had substantive argument (that indicated real malice on part of Deep Mind) it would have been easily published in Lancet, BMJ, even New England Journal of Medicine. Instead the fact its published in some subject specific Journal should tell you about concreteness of their "findings".

stuaxo 10y ago
This sort of thing is important in a context where more and more NHS services are getting sold off to private entities. Arrangements that have been in the place for decades in the US are not necessarily relevant to the UK.
- aub3bhat 10y ago
  "What sort of thing?" The report essentially says Google owns Deep Mind and Google has advertising business, ipso facto, some magical standrard dreamt up by the authors were not met. The reason for mentioning US agreements is that several nations, researchers and comapnies have developed procedures around sharing this data. And such sharing is not entirely unprecedented.
  - Silhouette 10y ago
    You're downplaying valid concerns. The ability to de-anonymise large data sets that we have today is unprecedented. The degree to which systems with access to that data are accessible remotely and potentially vulnerable to security problems is unprecedented. The degree to which powerful organisations like employers and insurers are attempting to profile potential employees and customers is unprecedented. However, neither major breaches involving huge amounts of potentially sensitive personal data nor scope creep in how data is used by large organisations once acquired are unprecedented. None of this is good for the individuals whose data is potentially at risk, and it's perfectly fair and reasonable to ask whether some of the most sensitive data there is about individuals is being properly handled by those entrusted with it given the implications of modern technology. There might be a lot of potential good in big data analysis for improving healthcare outcomes, but there is also a lot of potential harm if people stop, say, being confident in discussing potentially limiting conditions with their doctors, or reaching out for help with mental health issues because they're worried about confidentiality.

tomxor 10y ago

I don't want my privacy invaded as much as the next person, but arguing about broadness of medical history in this context is pretty stupid. The data needs to be broad if you are interested in finding out things you don't already know... that's why it's being fed into a machine learning algorithm in the first place - If you get selective then it's not going to be very useful - how do you limit history to what's relevant when you don't know whats relevant? There is however an interesting difference here between most data mining on the web which is trying to sell advertising. In this instance it should be fully anonymisable, only the doctor should be allowed to see that patient ID e76f57a is John Smith.

3JPLW 10y ago
Note that there's a big difference between de-identifying and fully anonymizing data. In general, it's extremely hard to fully anonymize a dataset. De-identification gets you most of the way there, but its frequently possible for a dedicated attacker to re-identify users by combining the data with public datasets. For example, patient ID e76f57b may be a 38 year old woman in Smalltown. Her medical record states that she gave birth on 3-20-2017. Find all public birth announcements from that day in that town with a 38 year old mother. Once you have a small set of candidates, it's not hard to narrow down farther.

akamaka 10y ago

You can skip reading the article, as it does not list any "errors" that have happened. It merely questions whether the agreement under which the data is shared has adequate protections.

3JPLW 10y ago
Indeed. The paper itself details seven "transgressions:" > 1) We do not know––and have no power to find out––what Google and DeepMind are really doing with NHS patient data, nor the extent of Royal Free’s meaningful control over what Google and DeepMind are doing; > 2) Any assurances about use of the dataset come from public relations statements, rather than independent oversight or legally binding documents; > 3) The amount of data transferred is far in excess of the requirements of those publicly stated needs, but not in excess of the information sharing agreement and broader memorandum of understanding governing the deal, both of which were kept private for many months; > 4) The data transfer was done without consulting relevant regulatory bodies, with only one superficial assessment of server security, combined with a post-hoc and inadequate privacy impact assessment; > 5) None of the millions of identified individuals in the dataset were either informed of the impending transfer to DeepMind, nor asked for their consent; > 6) The transfer relies on an argument that DeepMind is in a “direct care” relationship with each patient that has been admitted to Royal Free constituent hospitals, even though DeepMind is developing an app that will only conceivably be used in the treatment of one sixth of those individuals; and > 7) More than 12 months into the deal being made, no regulator had issued any comment or pushback. Quite a few of these strike me as rather absurd, but I don't know the regulatory environment in the UK.
- Silhouette 10y ago
  Most of these issues potentially involve direct violations of both standard practices within our healthcare regulatory framework and explicit data protection laws. They aren't absurd at all. They're talking about doing an end-run around our most fundamental privacy safeguards, in relation to some of the most sensitive personal data that exists about any given individual.
- throwaway729 10y ago

mikecb 10y ago

Interestingly, Deepmind's Ben Laurie has been working on certificate transparency-like tech to permit a verifiable audit log of access to data like this, precisely for this purpose: https://qz.com/929833/googles-goog-deepmind-is-using-blockchain-technology-to-handle-nhs-medical-data/ https://qz.com/929833/googles-goog-deepmind-is-using-blockch... If you want to take a look at the underlying technology, take a look at https://github.com/google/trillian https://github.com/google/trillian

KCFforecast 10y ago

Under Spain protection of personal data law, any file with personal information must be accessible for the person, and the person has the right to know, modify and delete that file, and to deny the access to that information for any purpose.

desas 10y ago
There must be qualifiers, you can't tell your bank to forget the money you owe it.
- KCFforecast 10y ago
  You are right, more information here: http://uk.practicallaw.com/1-520-8264 http://uk.practicallaw.com/1-520-8264

biggio 10y ago

I don't care or why should I care? As long as there's progress in making NHS more efficient in treating me that's fine. If they need it I will personally go to their offices and give blood samples or whatever they need on a daily basis!

Gatsky 10y ago

"In July 2015, clinicians from British public hospitals within the Royal Free London NHS Foundation Trust approached Google DeepMind Technologies Limited, an artificial intelligence company with no experience in providing healthcare services, about developing software using patient data from the Trust." Actually, the institution which collects and stores the data handed it over without due process. The article keeps trying to blame DeepMind, I guess criticising the NHS is a little stale. I think this is in a journal because it is too long for an op-ed but not substantial enough for a long form article.

killjoywashere 10y ago

I would urge folks writing these deals to make sure to separate the rights assigned to 1) algorithms, 2) data, and 3) the models they generate. There is a clear joint interest in the models, and that seems to get missed in most of these articles.