11 ms·

Download all of your GitHub data

151 points by TimWolla 8y ago

micky_t 8y ago
but I still cannot delete it...?
- judge2020 8y ago
  How would deleting work? They can't really anonymize any commit data without breaking git clones, so I guess deleting data would just delete anything non-git like PR text, issues, etc?
  - mehrdadn 8y ago
    Yeah, I think deleting issues etc. should be possible.
  - avar 8y ago
    There was a big discussion about this on the Git ML recently: https://public-inbox.org/git/20180417191549.GA1199@helen.PLASMA.Xg8.DE/ https://public-inbox.org/git/20180417191549.GA1199@helen.PLA... I.e. whether the argument that you're exempt from deleting data under the GDPR just because it's inconvenient for your storage format is going to fly in the EU. If not there's scary times ahead for Git. Here's a good example of a 4 year old using (presumably) her full name committing to the kernel: https://github.com/torvalds/linux/commit/690b0543a813b0ecfc51b0374c0ce6c8275435f0 https://github.com/torvalds/linux/commit/690b0543a813b0ecfc5... If that kid grows up, moves to the EU, and wants Microsoft (GitHub) to stop publishing that content, will they be able to just say "no"? How about if the kid has had their personal data embedded in the Bitcoin blockchain? Interesting times indeed.
    - craftyguy 8y ago
      In addition to making the request difficult for systems like git, if inconvenienced storage formats are exempt then this could easily be a loophole to circumvent the GDPR entirely. Facefriend, Instatwit, etc could, e.g., store all user data in something like git (your messages are committed using your name, etc) and point to this convoluted system as an excuse to not comply.
- fingerprinter 8y ago
  You've been able to permanently delete your GitHub profile and all data associated with it for quite some time. https://help.github.com/articles/deleting-your-user-account/ https://help.github.com/articles/deleting-your-user-account/ I've seen this 'ghost user' around on threads for years....
AdmiralAsshat 8y ago
Now what I would really like to see is some tool for keeping my GitHub repos in sync with my GitLab ones. I did a one-time migration from GH to GL, but my GH account is still open, primarily because some projects are GH-only, meaning I might continue to make forks, contributions, etc. Some way of making sure that I can periodically get all of those over to my GL would be nice.
- scrollaway 8y ago
  Gitlab itself supports being a mirror of an upstream GitHub remote (or any remote). Last I checked it doesn't deal well with force pushes but I haven't checked in a long time.
- kbenson 8y ago
  Not that I'm all that good at the more esoteric features of git, but you can overwrite the history with the correct flags, right? It seems like all you would need to do to sync them is have the most recent checkout of each, a way to determine which was newer, and push the newer branch over the older one (to be simplistic and destructive). If you're always overwriting history of one to make it match the other, diffing the git logs of each against each other would likely tell you which had items the other didn't, and if it went just one direction or both (i.e. can you just overwrite one with the other or is some manual intervention required).
  - MereInterest 8y ago
    The git repo itself is the easy part. You just make a local clone with two remote repos, pull from one, and push to the other. The hard part is all the parts that aren't managed by git, the issue tracker, PRs, releases, etc.
- sytse 8y ago
  Thanks for using GitLab! Please also see https://docs.gitlab.com/ee/workflow/repository_mirroring.html https://docs.gitlab.com/ee/workflow/repository_mirroring.htm... Pushing free to use, pulling is paid feature but public projects can use it for free too.
mohsen1 8y ago
Is this for GDPR compliance?
- ceejayoz 8y ago
  Almost certainly, but it's nice they're making it generally available. I've seen some sites' GDPR exports limited to EU customers, which seems like a dick move.
  - notimetorelax 8y ago
    The thing is, sites cannot use IP geolocation to decide if their users are EU residents or not. I think in the end it’s more expensive to try to manage this than to let everyone do the export and deletion.
    - sb8244 8y ago
      Don't you just need to provide access to the data within 30 days of asking for it? That would allow for not making it public but having it publicly available.
      - notimetorelax 8y ago
        I’m not a GDPR lawyer, but AFAIK export has to be satisfied within 7 days and removal within 30 days. Not sure what are the delays for a correction. I think the most expensive part could be the human time needed to make judgments on the residence of the requestor. Fully automated processes available to all the user based won’t have this extra cost.
  - Someone1234 8y ago
    Reminds me of California's labels in that way. Everyone mocks them, but some of them are legitimately useful e.g.: https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-couch-chemicals-just-look-new-label https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-c... Same thing here, because border specific stuff aren't worth the hassle, the lowest common denominator (in this case GDPR) becomes the new international standard. Additionally some US politicians have proposed copying parts of the GDPR into US law which would pretty much make this a permanent staple.
eikenberry 8y ago
Anyone know if it includes information on which repositories were given stars?
- TimWolla 8y ago
  It does not. This is what it looks like for me (files with number > 1 omitted for brevity): -rw-r--r-- 1 timwolla timwolla 499 Dec 19 19:31 attachments_000001.json -rw-r--r-- 1 timwolla timwolla 52K Dec 19 19:31 commit_comments_000001.json -rw-r--r-- 1 timwolla timwolla 72K Dec 19 19:31 issue_comments_000001.json -rw-r--r-- 1 timwolla timwolla 35K Dec 19 19:31 issue_events_000001.json -rw-r--r-- 1 timwolla timwolla 34K Dec 19 19:31 issues_000001.json -rw-r--r-- 1 timwolla timwolla 356 Dec 19 19:31 milestones_000001.json -rw-r--r-- 1 timwolla timwolla 22K Dec 19 19:31 projects_000001.json -rw-r--r-- 1 timwolla timwolla 4.4K Dec 19 19:31 protected_branches_000001.json -rw-r--r-- 1 timwolla timwolla 25K Dec 19 19:31 pull_request_review_comments_000001.json -rw-r--r-- 1 timwolla timwolla 4.3K Dec 19 19:31 pull_request_reviews_000001.json -rw-r--r-- 1 timwolla timwolla 51K Dec 19 19:31 pull_requests_000001.json -rw-r--r-- 1 timwolla timwolla 9.0K Dec 19 19:31 releases_000001.json -rw-r--r-- 1 timwolla timwolla 80K Dec 19 19:30 repositories_000001.json -rw-r--r-- 1 timwolla timwolla 75 Dec 19 19:31 schema.json -rw-r--r-- 1 timwolla timwolla 27K Dec 19 19:30 users_000001.json attachments: total 12K drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:31 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 c544f416-10c9-11e7-8337-82f88cac9fcd/ releases: total 36K drwxr-xr-x 9 timwolla timwolla 4.0K Dec 19 19:31 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 5aa3a138-9c86-11e5-99bc-79a6c3e509eb/ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 … drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 dba26980-d9f1-11e8-8357-3f2750c60e51/ repositories: total 12K drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:30 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 41 timwolla timwolla 4.0K Dec 19 19:31 TimWolla/
3pt14159 8y ago
I want to be able to freeze this. Give me 30 days to OK a bulk data download. I don't want a single lost access token to have such an easy way for someone to get everything.
- driverdan 8y ago
  A single lost access token + the API is an easy way to get everything too. It doesn't take much code to get everything that way.
  - 3pt14159 8y ago
    Well then I want that to be locked down too! I'm sick of the all-or-nothingness of many of these security measures.
    - zrail 8y ago
      Really the lesson is to use constrained API tokens, going so far as to make single-purpose users as necessary.
    - SmellyGeekBoy 8y ago
      Then don't create an API token with full access?
- akerl_ 8y ago
  A single lost access token that had global permissions already had the ability to get everything: GitHub’s API is already very full-featured. The fix here is to use their scoping as intended: it should be amazingly rare to need an access token with such broad access.
- benbalter 8y ago
  When you request an archive of your data, we send the download link to your primary email address (the required token is not available via the web UI). Once you click that link, you'll be asked to re-enter your password. So for this particular feature, an attacker would need both your GitHub password (and your 2FA seed or an active session if 2FA is enabled) and access to your email.
ocdtrekkie 8y ago
Glad to see more sites jumping on this party, I just wish more tools were out there that interpreted data from these dumps directly and made them more readable. I've started the practice of keeping backups of somewhat recent "all my data" dumps from each site that offers them.
afezelv2vk 8y ago
It seems odd for GitHub to be jumping through GDPR hoops - while at the same time RETROACTIVELY revealing all force-pushes [aka previously hidden and irretrievable by clone] in pull requests. Of course, that data was compromised the second it was pushed, but, now they're parading the fact that people make mistakes... What side are you on GitHub? Privacy? Or not?
- saagarjha 8y ago
  Force pushing code doesn’t make your old code private.
  - Sytten 8y ago
    That's an interesting point though, if you made the mistake of publishing a private key at some point and force-pushed your repo to remove it you expect it to be gone from the system, but it's not. I wonder if you could actually retrieve it from the cloned repo. Is it hidden somewhere in git? EDIT: The cloned repos seems good, but following this post: https://stackoverflow.com/questions/3973994/how-can-i-recover-from-an-erronous-git-push-f-origin-master https://stackoverflow.com/questions/3973994/how-can-i-recove... you can easily find some "lost" commit and retrieve secrets. I checked with one of my repo and I realized I had compromised information...
Sytten 8y ago
Legit question: what about the data stored about you, but that you have no longer access to? I have no clue if that is also covered by GDPR?
stilist 8y ago
It wasn't clear to me from the blog post, but the data for events, issues, attachments, and so on is only available for repositories you own. Activity in repos belonging to organizations or other users isn't included.

151 points by TimWolla 8y ago

micky_t 8y ago

but I still cannot delete it...?

judge2020 8y ago
How would deleting work? They can't really anonymize any commit data without breaking git clones, so I guess deleting data would just delete anything non-git like PR text, issues, etc?
- mehrdadn 8y ago
  Yeah, I think deleting issues etc. should be possible.
- avar 8y ago
  There was a big discussion about this on the Git ML recently: https://public-inbox.org/git/20180417191549.GA1199@helen.PLASMA.Xg8.DE/ https://public-inbox.org/git/20180417191549.GA1199@helen.PLA... I.e. whether the argument that you're exempt from deleting data under the GDPR just because it's inconvenient for your storage format is going to fly in the EU. If not there's scary times ahead for Git. Here's a good example of a 4 year old using (presumably) her full name committing to the kernel: https://github.com/torvalds/linux/commit/690b0543a813b0ecfc51b0374c0ce6c8275435f0 https://github.com/torvalds/linux/commit/690b0543a813b0ecfc5... If that kid grows up, moves to the EU, and wants Microsoft (GitHub) to stop publishing that content, will they be able to just say "no"? How about if the kid has had their personal data embedded in the Bitcoin blockchain? Interesting times indeed.
  - craftyguy 8y ago
    In addition to making the request difficult for systems like git, if inconvenienced storage formats are exempt then this could easily be a loophole to circumvent the GDPR entirely. Facefriend, Instatwit, etc could, e.g., store all user data in something like git (your messages are committed using your name, etc) and point to this convoluted system as an excuse to not comply.
fingerprinter 8y ago
You've been able to permanently delete your GitHub profile and all data associated with it for quite some time. https://help.github.com/articles/deleting-your-user-account/ https://help.github.com/articles/deleting-your-user-account/ I've seen this 'ghost user' around on threads for years....

AdmiralAsshat 8y ago

Now what I would really like to see is some tool for keeping my GitHub repos in sync with my GitLab ones. I did a one-time migration from GH to GL, but my GH account is still open, primarily because some projects are GH-only, meaning I might continue to make forks, contributions, etc. Some way of making sure that I can periodically get all of those over to my GL would be nice.

scrollaway 8y ago
Gitlab itself supports being a mirror of an upstream GitHub remote (or any remote). Last I checked it doesn't deal well with force pushes but I haven't checked in a long time.
kbenson 8y ago
Not that I'm all that good at the more esoteric features of git, but you can overwrite the history with the correct flags, right? It seems like all you would need to do to sync them is have the most recent checkout of each, a way to determine which was newer, and push the newer branch over the older one (to be simplistic and destructive). If you're always overwriting history of one to make it match the other, diffing the git logs of each against each other would likely tell you which had items the other didn't, and if it went just one direction or both (i.e. can you just overwrite one with the other or is some manual intervention required).
- MereInterest 8y ago
  The git repo itself is the easy part. You just make a local clone with two remote repos, pull from one, and push to the other. The hard part is all the parts that aren't managed by git, the issue tracker, PRs, releases, etc.
sytse 8y ago
Thanks for using GitLab! Please also see https://docs.gitlab.com/ee/workflow/repository_mirroring.html https://docs.gitlab.com/ee/workflow/repository_mirroring.htm... Pushing free to use, pulling is paid feature but public projects can use it for free too.

mohsen1 8y ago

Is this for GDPR compliance?

ceejayoz 8y ago
Almost certainly, but it's nice they're making it generally available. I've seen some sites' GDPR exports limited to EU customers, which seems like a dick move.
- notimetorelax 8y ago
  The thing is, sites cannot use IP geolocation to decide if their users are EU residents or not. I think in the end it’s more expensive to try to manage this than to let everyone do the export and deletion.
  - sb8244 8y ago
    Don't you just need to provide access to the data within 30 days of asking for it? That would allow for not making it public but having it publicly available.
    - notimetorelax 8y ago
      I’m not a GDPR lawyer, but AFAIK export has to be satisfied within 7 days and removal within 30 days. Not sure what are the delays for a correction. I think the most expensive part could be the human time needed to make judgments on the residence of the requestor. Fully automated processes available to all the user based won’t have this extra cost.
- Someone1234 8y ago
  Reminds me of California's labels in that way. Everyone mocks them, but some of them are legitimately useful e.g.: https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-couch-chemicals-just-look-new-label https://www.nrdc.org/experts/veena-singla/want-avoid-toxic-c... Same thing here, because border specific stuff aren't worth the hassle, the lowest common denominator (in this case GDPR) becomes the new international standard. Additionally some US politicians have proposed copying parts of the GDPR into US law which would pretty much make this a permanent staple.

eikenberry 8y ago

Anyone know if it includes information on which repositories were given stars?

TimWolla 8y ago
It does not. This is what it looks like for me (files with number > 1 omitted for brevity): -rw-r--r-- 1 timwolla timwolla 499 Dec 19 19:31 attachments_000001.json -rw-r--r-- 1 timwolla timwolla 52K Dec 19 19:31 commit_comments_000001.json -rw-r--r-- 1 timwolla timwolla 72K Dec 19 19:31 issue_comments_000001.json -rw-r--r-- 1 timwolla timwolla 35K Dec 19 19:31 issue_events_000001.json -rw-r--r-- 1 timwolla timwolla 34K Dec 19 19:31 issues_000001.json -rw-r--r-- 1 timwolla timwolla 356 Dec 19 19:31 milestones_000001.json -rw-r--r-- 1 timwolla timwolla 22K Dec 19 19:31 projects_000001.json -rw-r--r-- 1 timwolla timwolla 4.4K Dec 19 19:31 protected_branches_000001.json -rw-r--r-- 1 timwolla timwolla 25K Dec 19 19:31 pull_request_review_comments_000001.json -rw-r--r-- 1 timwolla timwolla 4.3K Dec 19 19:31 pull_request_reviews_000001.json -rw-r--r-- 1 timwolla timwolla 51K Dec 19 19:31 pull_requests_000001.json -rw-r--r-- 1 timwolla timwolla 9.0K Dec 19 19:31 releases_000001.json -rw-r--r-- 1 timwolla timwolla 80K Dec 19 19:30 repositories_000001.json -rw-r--r-- 1 timwolla timwolla 75 Dec 19 19:31 schema.json -rw-r--r-- 1 timwolla timwolla 27K Dec 19 19:30 users_000001.json attachments: total 12K drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:31 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 c544f416-10c9-11e7-8337-82f88cac9fcd/ releases: total 36K drwxr-xr-x 9 timwolla timwolla 4.0K Dec 19 19:31 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 5aa3a138-9c86-11e5-99bc-79a6c3e509eb/ drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 … drwxr-xr-x 2 timwolla timwolla 4.0K Dec 19 19:31 dba26980-d9f1-11e8-8357-3f2750c60e51/ repositories: total 12K drwxr-xr-x 3 timwolla timwolla 4.0K Dec 19 19:30 ./ drwxr-xr-x 5 timwolla timwolla 4.0K Dec 19 19:31 ../ drwxr-xr-x 41 timwolla timwolla 4.0K Dec 19 19:31 TimWolla/

3pt14159 8y ago

I want to be able to freeze this. Give me 30 days to OK a bulk data download. I don't want a single lost access token to have such an easy way for someone to get everything.

driverdan 8y ago
A single lost access token + the API is an easy way to get everything too. It doesn't take much code to get everything that way.
- 3pt14159 8y ago
  Well then I want that to be locked down too! I'm sick of the all-or-nothingness of many of these security measures.
  - zrail 8y ago
    Really the lesson is to use constrained API tokens, going so far as to make single-purpose users as necessary.
  - SmellyGeekBoy 8y ago
    Then don't create an API token with full access?
akerl_ 8y ago
A single lost access token that had global permissions already had the ability to get everything: GitHub’s API is already very full-featured. The fix here is to use their scoping as intended: it should be amazingly rare to need an access token with such broad access.
benbalter 8y ago
When you request an archive of your data, we send the download link to your primary email address (the required token is not available via the web UI). Once you click that link, you'll be asked to re-enter your password. So for this particular feature, an attacker would need both your GitHub password (and your 2FA seed or an active session if 2FA is enabled) and access to your email.

ocdtrekkie 8y ago

Glad to see more sites jumping on this party, I just wish more tools were out there that interpreted data from these dumps directly and made them more readable. I've started the practice of keeping backups of somewhat recent "all my data" dumps from each site that offers them.

afezelv2vk 8y ago

It seems odd for GitHub to be jumping through GDPR hoops - while at the same time RETROACTIVELY revealing all force-pushes [aka previously hidden and irretrievable by clone] in pull requests. Of course, that data was compromised the second it was pushed, but, now they're parading the fact that people make mistakes... What side are you on GitHub? Privacy? Or not?

saagarjha 8y ago
Force pushing code doesn’t make your old code private.
- Sytten 8y ago
  That's an interesting point though, if you made the mistake of publishing a private key at some point and force-pushed your repo to remove it you expect it to be gone from the system, but it's not. I wonder if you could actually retrieve it from the cloned repo. Is it hidden somewhere in git? EDIT: The cloned repos seems good, but following this post: https://stackoverflow.com/questions/3973994/how-can-i-recover-from-an-erronous-git-push-f-origin-master https://stackoverflow.com/questions/3973994/how-can-i-recove... you can easily find some "lost" commit and retrieve secrets. I checked with one of my repo and I realized I had compromised information...

Sytten 8y ago

Legit question: what about the data stored about you, but that you have no longer access to? I have no clue if that is also covered by GDPR?

stilist 8y ago

It wasn't clear to me from the blog post, but the data for events, issues, attachments, and so on is only available for repositories you own. Activity in repos belonging to organizations or other users isn't included.