9 ms·

UK Biobank health data keeps ending up on GitHub

197 points by Cynddl 5mo ago

I'm a researcher studying privacy, and I started tracking the DMCA notices that UK Biobank sends to GitHub. I tracked 110 notices filed so far, targeting 197 code repositories by 170 developers across the world. The exposure of Biobank data on GitHub is the latest in a long series of governance challenges for UK Biobank. (My colleague and I have an editorial in the BMJ about this: http://bmj.com/cgi/content/full/bmj.s660?ijkey=dEot4dJZGZGXeG1&keytype=ref http://bmj.com/cgi/content/full/bmj.s660?ijkey=dEot4dJZGZGXe...). The latest is today, with information of all half a million members listed for sale on Alibaba. Looking at the takedown notices, we often see specific files being targeted rather than entire repositories (possibly to justify the copyright infringement as required for a takedown notice, not a copyright expert; although it is clear that they only use DMCA notices as a last resort, for GitHub users they cannot identify, and who were likely not given access in the first place). A quarter of the files are genetic/genomics. Tabular data account for another large share and could contain phenotype or health records.

mil22 5mo ago
The irony is, they don’t even provide the data to the participants themselves.
- vain 5mo ago
  Huh? I got my report over email. I think you have to ask for it.
michaelt 5mo ago
> It has given 20,000 researchers around the world access under strict agreements that prohibit sharing data further. To me it seems rather naive to have done that. After all, you can't un-leak medical data. So even if the "strict agreement" included huge punishments, there's no getting the toothpaste back in the tube. If you want to ensure compliance before a leak happens you have to (ugh) audit their compliance. And that isn't something that scales to 20,000 researchers. Too late to do anything about it now though :(
- SilverElfin 5mo ago
  That’s insane. And what does researcher even mean - some random university student? What would they know about securing that data? I wonder if the people whose data is out there even know this is happening
  - 7bees 5mo ago
    The people involved are volunteers. The rules for getting access are readily available, and clearly don't include "some random university student": https://www.ukbiobank.ac.uk/about-us/how-we-work/access-to-uk-biobank-data/ https://www.ukbiobank.ac.uk/about-us/how-we-work/access-to-u...
    - siva7 5mo ago
      They clearly do include "some random student" as the data can be shared with others from the eligible research group which are almost always university students who have zero clue about itsec.
      - globular-toast 5mo ago
        I worked in this field. It's not just the students. Hardly anyone seemed to understand how and why you would keep data out of a git repo.
        nxobject 5mo ago
        I'm curious – in which context? I've worked on NIH-funded grants in academic medical centers, throughout the research lifecycle, and I've seen how both stringently data management plans are vetted, and how annual IRB certification drills the basics even into the oldest tech-phobic investigators. That being said, I may be as pessmistic as you are: I don't think people right now grasp how standards for deidentification may no longer be enough, and how easy and automated deanonymization changes everything. Unfortunately, cuts to federal science agencies means that I doubt any well-informed guidance will come soon.
John7878781 5mo ago
What are the pros/cons of just open-sourcing everything for future bio bank projects?
- Cynddl 5mo ago
  You mean giving anyone access to the data? Or open sourcing the code? If the latter, I think that's a generally a good practice. Security through obscurity is never good for public infrastructure. In this case, UK Biobank has now switched to a remote access platform (not particularly secure, as the data was found for sale on Alibaba today), but contracting it to DNAnexus and Amazon. Private companies have no incentives to open source data, unless mandated to do so. In the EU, there is a bigger interest in building scalable but also secure platforms for health data. Hopefully good innovation will come from there.
- renewiltord 5mo ago
  Hard to do. The same people with the collection and tracking infrastructure required are infinitely sue-able so you need legal protection if anything goes wrong.
  - ashley95 5mo ago
    Really don't think this is any issue given the post we are commenting on...
- culi 5mo ago
  The people who agreed to contribute their biodata did not consent to that. If you want such a project you need to have a new project with a different agreement. I doubt you could get as many volunteers to freely give away such intimate data to anyone who wants though
- michaelt 5mo ago
  It's exceptionally difficult to avoid the data being de-anonymised. If an 'anonymised' medical record says the person was born 6th September 1969, received treatment for a broken arm on 1 April 2004, and received a course of treatment in 2009 after catching the clap on holiday in Thailand - that's enough bits of information to uniquely identify me. And medical researchers are usually very big on 'fully informed consent' so they can't gloss over that reality, hide it in fine print or obsfucate it with flowerly language. They usually have to make sure the participants really understand what they're agreeing to. It might still work out fine, of course - 95% of people's medical histories don't contain anything particularly embarrassing, so you might be able to get plenty of participants anyway.
captn3m0 5mo ago
Took me 5 minutes to find more: https://github.com/tanaylab/Mendelson_et_al_2023/blob/9c5a653f6025506c984ee91d528525aded2022f2/Disease_Longevity_UKBB.ipynb#L72 https://github.com/tanaylab/Mendelson_et_al_2023/blob/9c5a65... (Uses Date of Birth column). And some information on how they were distributing it to researchers: https://github.com/broadinstitute/ml4h/blob/master/ingest/ukbb_csv_bigquery/README.md https://github.com/broadinstitute/ml4h/blob/master/ingest/uk... > The following steps require the ukbunpack and ukbconv utilities from the UK Biobank website. The file decrypt_all.sh will run through the following steps on one of the on-prem servers. > Once the data is downloaded, it needs to be "ukbunpacked" which decrypts it, and then converts it to a file format of choice. Both ukbunpack and ukbconv are available from the UK Biobank's website. The decryption has to happen on a linux system if you download the linux tools, e.g. the Broad's on-prem servers. Note that you need plenty of space to decrypt/unpack, and the programs may fail silently if disk space runs out during the middle. https://biobank.ctsu.ox.ac.uk/crystal/download.cgi https://biobank.ctsu.ox.ac.uk/crystal/download.cgi
- Cynddl 5mo ago
  Good catch! The data is everywhere, re-uploaded every week. I am aware of ~30 repositories that UK Biobank has asked GitHub to delete, and can still be found elsewhere online. They know the site, they have managed to delete data from that site before, and yet the files are still there.
- nebmfield 5mo ago
  [dead]
- philipwhiuk 5mo ago
  I don't think either of those links contain actual PII (or anonymised PII). (The first is a GitHub repo for https://www.weizmann.ac.il/math/tanay/home https://www.weizmann.ac.il/math/tanay/home )
adwf 5mo ago
That's the least of it: https://www.bbc.co.uk/news/articles/cpvxgl3n138o https://www.bbc.co.uk/news/articles/cpvxgl3n138o All 500,000 participants for sale on Alibaba... And official response: https://www.ukbiobank.ac.uk/news/a-message-to-our-participants-uk-biobank-data-security-update/ https://www.ukbiobank.ac.uk/news/a-message-to-our-participan...
- fastaguy88 5mo ago
  BioBank claims (1) only de-identified data was available and (2) none of the data was actually sold before the datasets were taken down.
  - john_strinlai 5mo ago
    unfortunately for most people, de-identified data is typically a very short analysis away from being re-identified. the field of de-anonymization is booming.
    - nxobject 5mo ago
      Especially for a nation-state that's already hoovering up data broker products.
- nxobject 5mo ago
  More details on leaked information from El Reg, especially after (laudably) the British government has been more transparent: https://www.theregister.com/2026/04/23/500k_biobank_volunteers_data_listed/ https://www.theregister.com/2026/04/23/500k_biobank_voluntee... "The charity did not specify the types of data that were included, but Murray stated in the Commons that several markers were included in the listings: - Gender - Age - Month and year of birth - Assessment center data - Attendance dates - Socioeconomic status - Lifestyle habits - Measures from biological samples related to haematology, biology, and chemistry - Sleep, diet, work environment, mental health, and health outcomes data."
- aaron695 5mo ago
  [dead]
nxobject 5mo ago
From the perspective of someone who's worked with (biostatisticians who touch) Medicaid and Medicare billing data... It looks like they've identified the institutions, at least... but aren't identifying it to the public for now. Are there going to be consequences? Are they going to be identified and sanctioned beyond "having their access suspended?" In the US, HHS wouldn't hestitate to name, shame, and impose a sanction with corrective action plans. Not knowing much about how things work across the pond, I'm sure CMS PII gets used more often in research without these leaks left and right.
anitil 5mo ago
I've opted in to Australia's version of the biobank knowing that it's inevitable that it will be leaked some day, I think the data is so valuable in perpetuity that it's worth it. I remember Ben Goldacre has been working on how to make data more accessible in a safer way to (in part) avoid this very thing, but I haven't heard much of it since [0] [0] https://www.bennett.ox.ac.uk/blog/2025/02/opensafely-in-brief/ https://www.bennett.ox.ac.uk/blog/2025/02/opensafely-in-brie...
- nxobject 5mo ago
  I like their idea of an audit log of analysis runs -- beyond transparency, I'm sure it'll help future researchers know how much iteration is needed to work with the messiness of medical records... I'm also amused (in a good way) by the fact that SAS isn't supported as an analysis platform...
  - anitil 5mo ago
    It's certainly an interesting idea, I remember he was on a few podcasts talking about it. I might submit it here to see if it gets some conversation going
- mfld 5mo ago
  This is the right mindset. Securing huge piles of heterogeneous data while giving PhD students the freedom to "play" with it are quite conflicting goals.
- philipwhiuk 5mo ago
  I have huge problems with Goldacre's project because that project has never been disclosed to the general public let alone some form of opt-in/opt-out.
  - harvey9 5mo ago
    I think the opt out process is the same as the one that came to prominence in the care.data days. It is not a very good process since it is opt out not opt in, and reliant on motivated people searching for a form. On the other hand I like opensafely's approach to security: no individual data is ever shared with researchers.
_alphageek 5mo ago
[dead]
dariosalvi78 5mo ago
the issue is with jupyter notebooks because they keep some of the data in the output (typically a few rows, but still). They should strongly recommend to use regular python scripts, and keep the jupyter books just for verification, which is a very sane thing to do also from a SW engineering perspective.
- paulwetzel 5mo ago
  I cant really understand why Jupyter Notebooks do this in the first place. It makes it (a) really hard to version control, as there will always be some random blob of non-textual data in the notebook that pops up in a diff and makes it basically unreadable and (b) I can't really see the benefit, as it only stores some part of the data, and not the full table, as far as I am aware. Enforcing Jupytext is a good adaption, and gives you all the, arguably really nice, comfort from a notebook, and the proper code practice from SW engineering.
- lmc 5mo ago
  marimo notebooks give you the best of both worlds (https://marimo.io https://marimo.io)
deknos 5mo ago
so.. where can i get this data? XD
NGRhodes 5mo ago
Thank you for sharing. I work in a central RSE team and have raised this topic to the team, with a view of bringing attention to this issue and better educating our researchers (as part of our training offerings and documentation).
mhh__ 5mo ago
I haven't been paying attention to it but wasn't there some kerfuffle over some people threatening to leak it over not being allowed to publish controversial findings?
anordal 5mo ago
Have medical researchers not heard of gitignore? Any hypothesis about the mechanism here?
khelavastr 5mo ago
UK government fails to prosecute criminally negligent software developers..

197 points by Cynddl 5mo ago

mil22 5mo ago

The irony is, they don’t even provide the data to the participants themselves.

vain 5mo ago
Huh? I got my report over email. I think you have to ask for it.

michaelt 5mo ago

> It has given 20,000 researchers around the world access under strict agreements that prohibit sharing data further. To me it seems rather naive to have done that. After all, you can't un-leak medical data. So even if the "strict agreement" included huge punishments, there's no getting the toothpaste back in the tube. If you want to ensure compliance before a leak happens you have to (ugh) audit their compliance. And that isn't something that scales to 20,000 researchers. Too late to do anything about it now though :(

SilverElfin 5mo ago
That’s insane. And what does researcher even mean - some random university student? What would they know about securing that data? I wonder if the people whose data is out there even know this is happening
- 7bees 5mo ago
  The people involved are volunteers. The rules for getting access are readily available, and clearly don't include "some random university student": https://www.ukbiobank.ac.uk/about-us/how-we-work/access-to-uk-biobank-data/ https://www.ukbiobank.ac.uk/about-us/how-we-work/access-to-u...
  - siva7 5mo ago
    They clearly do include "some random student" as the data can be shared with others from the eligible research group which are almost always university students who have zero clue about itsec.
    - globular-toast 5mo ago
      I worked in this field. It's not just the students. Hardly anyone seemed to understand how and why you would keep data out of a git repo.
      - nxobject 5mo ago
        I'm curious – in which context? I've worked on NIH-funded grants in academic medical centers, throughout the research lifecycle, and I've seen how both stringently data management plans are vetted, and how annual IRB certification drills the basics even into the oldest tech-phobic investigators. That being said, I may be as pessmistic as you are: I don't think people right now grasp how standards for deidentification may no longer be enough, and how easy and automated deanonymization changes everything. Unfortunately, cuts to federal science agencies means that I doubt any well-informed guidance will come soon.

John7878781 5mo ago

What are the pros/cons of just open-sourcing everything for future bio bank projects?

Cynddl 5mo ago
You mean giving anyone access to the data? Or open sourcing the code? If the latter, I think that's a generally a good practice. Security through obscurity is never good for public infrastructure. In this case, UK Biobank has now switched to a remote access platform (not particularly secure, as the data was found for sale on Alibaba today), but contracting it to DNAnexus and Amazon. Private companies have no incentives to open source data, unless mandated to do so. In the EU, there is a bigger interest in building scalable but also secure platforms for health data. Hopefully good innovation will come from there.
renewiltord 5mo ago
Hard to do. The same people with the collection and tracking infrastructure required are infinitely sue-able so you need legal protection if anything goes wrong.
- ashley95 5mo ago
  Really don't think this is any issue given the post we are commenting on...
culi 5mo ago
The people who agreed to contribute their biodata did not consent to that. If you want such a project you need to have a new project with a different agreement. I doubt you could get as many volunteers to freely give away such intimate data to anyone who wants though
michaelt 5mo ago
It's exceptionally difficult to avoid the data being de-anonymised. If an 'anonymised' medical record says the person was born 6th September 1969, received treatment for a broken arm on 1 April 2004, and received a course of treatment in 2009 after catching the clap on holiday in Thailand - that's enough bits of information to uniquely identify me. And medical researchers are usually very big on 'fully informed consent' so they can't gloss over that reality, hide it in fine print or obsfucate it with flowerly language. They usually have to make sure the participants really understand what they're agreeing to. It might still work out fine, of course - 95% of people's medical histories don't contain anything particularly embarrassing, so you might be able to get plenty of participants anyway.

captn3m0 5mo ago

Took me 5 minutes to find more: https://github.com/tanaylab/Mendelson_et_al_2023/blob/9c5a653f6025506c984ee91d528525aded2022f2/Disease_Longevity_UKBB.ipynb#L72 https://github.com/tanaylab/Mendelson_et_al_2023/blob/9c5a65... (Uses Date of Birth column). And some information on how they were distributing it to researchers: https://github.com/broadinstitute/ml4h/blob/master/ingest/ukbb_csv_bigquery/README.md https://github.com/broadinstitute/ml4h/blob/master/ingest/uk... > The following steps require the ukbunpack and ukbconv utilities from the UK Biobank website. The file decrypt_all.sh will run through the following steps on one of the on-prem servers. > Once the data is downloaded, it needs to be "ukbunpacked" which decrypts it, and then converts it to a file format of choice. Both ukbunpack and ukbconv are available from the UK Biobank's website. The decryption has to happen on a linux system if you download the linux tools, e.g. the Broad's on-prem servers. Note that you need plenty of space to decrypt/unpack, and the programs may fail silently if disk space runs out during the middle. https://biobank.ctsu.ox.ac.uk/crystal/download.cgi https://biobank.ctsu.ox.ac.uk/crystal/download.cgi

Cynddl 5mo ago
Good catch! The data is everywhere, re-uploaded every week. I am aware of ~30 repositories that UK Biobank has asked GitHub to delete, and can still be found elsewhere online. They know the site, they have managed to delete data from that site before, and yet the files are still there.
nebmfield 5mo ago
[dead]
philipwhiuk 5mo ago
I don't think either of those links contain actual PII (or anonymised PII). (The first is a GitHub repo for https://www.weizmann.ac.il/math/tanay/home https://www.weizmann.ac.il/math/tanay/home )

adwf 5mo ago

That's the least of it: https://www.bbc.co.uk/news/articles/cpvxgl3n138o https://www.bbc.co.uk/news/articles/cpvxgl3n138o All 500,000 participants for sale on Alibaba... And official response: https://www.ukbiobank.ac.uk/news/a-message-to-our-participants-uk-biobank-data-security-update/ https://www.ukbiobank.ac.uk/news/a-message-to-our-participan...

fastaguy88 5mo ago
BioBank claims (1) only de-identified data was available and (2) none of the data was actually sold before the datasets were taken down.
- john_strinlai 5mo ago
  unfortunately for most people, de-identified data is typically a very short analysis away from being re-identified. the field of de-anonymization is booming.
  - nxobject 5mo ago
    Especially for a nation-state that's already hoovering up data broker products.
nxobject 5mo ago
More details on leaked information from El Reg, especially after (laudably) the British government has been more transparent: https://www.theregister.com/2026/04/23/500k_biobank_volunteers_data_listed/ https://www.theregister.com/2026/04/23/500k_biobank_voluntee... "The charity did not specify the types of data that were included, but Murray stated in the Commons that several markers were included in the listings: - Gender - Age - Month and year of birth - Assessment center data - Attendance dates - Socioeconomic status - Lifestyle habits - Measures from biological samples related to haematology, biology, and chemistry - Sleep, diet, work environment, mental health, and health outcomes data."
aaron695 5mo ago
[dead]

nxobject 5mo ago

From the perspective of someone who's worked with (biostatisticians who touch) Medicaid and Medicare billing data... It looks like they've identified the institutions, at least... but aren't identifying it to the public for now. Are there going to be consequences? Are they going to be identified and sanctioned beyond "having their access suspended?" In the US, HHS wouldn't hestitate to name, shame, and impose a sanction with corrective action plans. Not knowing much about how things work across the pond, I'm sure CMS PII gets used more often in research without these leaks left and right.

anitil 5mo ago

I've opted in to Australia's version of the biobank knowing that it's inevitable that it will be leaked some day, I think the data is so valuable in perpetuity that it's worth it. I remember Ben Goldacre has been working on how to make data more accessible in a safer way to (in part) avoid this very thing, but I haven't heard much of it since [0] [0] https://www.bennett.ox.ac.uk/blog/2025/02/opensafely-in-brief/ https://www.bennett.ox.ac.uk/blog/2025/02/opensafely-in-brie...

nxobject 5mo ago
I like their idea of an audit log of analysis runs -- beyond transparency, I'm sure it'll help future researchers know how much iteration is needed to work with the messiness of medical records... I'm also amused (in a good way) by the fact that SAS isn't supported as an analysis platform...
- anitil 5mo ago
  It's certainly an interesting idea, I remember he was on a few podcasts talking about it. I might submit it here to see if it gets some conversation going
mfld 5mo ago
This is the right mindset. Securing huge piles of heterogeneous data while giving PhD students the freedom to "play" with it are quite conflicting goals.
philipwhiuk 5mo ago
I have huge problems with Goldacre's project because that project has never been disclosed to the general public let alone some form of opt-in/opt-out.
- harvey9 5mo ago
  I think the opt out process is the same as the one that came to prominence in the care.data days. It is not a very good process since it is opt out not opt in, and reliant on motivated people searching for a form. On the other hand I like opensafely's approach to security: no individual data is ever shared with researchers.

_alphageek 5mo ago

[dead]

dariosalvi78 5mo ago

the issue is with jupyter notebooks because they keep some of the data in the output (typically a few rows, but still). They should strongly recommend to use regular python scripts, and keep the jupyter books just for verification, which is a very sane thing to do also from a SW engineering perspective.

paulwetzel 5mo ago
I cant really understand why Jupyter Notebooks do this in the first place. It makes it (a) really hard to version control, as there will always be some random blob of non-textual data in the notebook that pops up in a diff and makes it basically unreadable and (b) I can't really see the benefit, as it only stores some part of the data, and not the full table, as far as I am aware. Enforcing Jupytext is a good adaption, and gives you all the, arguably really nice, comfort from a notebook, and the proper code practice from SW engineering.
lmc 5mo ago
marimo notebooks give you the best of both worlds (https://marimo.io https://marimo.io)

deknos 5mo ago

so.. where can i get this data? XD

NGRhodes 5mo ago

Thank you for sharing. I work in a central RSE team and have raised this topic to the team, with a view of bringing attention to this issue and better educating our researchers (as part of our training offerings and documentation).

mhh__ 5mo ago

I haven't been paying attention to it but wasn't there some kerfuffle over some people threatening to leak it over not being allowed to publish controversial findings?

anordal 5mo ago

Have medical researchers not heard of gitignore? Any hypothesis about the mechanism here?

khelavastr 5mo ago

UK government fails to prosecute criminally negligent software developers..