3 ms·

Github Archive

38 points by ig1 14y ago

danso 14y ago
Dumb question, but is the code behind the archive open source? I didn't see any link but did see the star button which... Ok...I see the star button serves as a link https://github.com/igrigorik/githubarchive.org/ https://github.com/igrigorik/githubarchive.org/ It's great to have other crawler projects to learn from
- briandoll 14y ago
  Yep, the repository is public and a big chunk of this public data is accessible on BigQuery for anyone to query and analyze. Ilya Grigorik and myself just gave a talk at Strata[1] on interesting analysis and visualizations that people did with the GitHub Archive as part of the GitHub data challenge[2]. We'll post a link to the talk soon. We announced during the talk that BigQuery will soon (in the next day or two) have an archive of the GitHub public timeline from February 2011 to now. I can't wait to see what people do with it! [1] http://strataconf.com/stratany2012/public/schedule/speaker/25018 http://strataconf.com/stratany2012/public/schedule/speaker/2... [2] https://github.com/blog/1162-github-data-challenge-winners https://github.com/blog/1162-github-data-challenge-winners
- PanMan 14y ago
  It is a great project. However, it's not based on crawling, as far as I understand: Github pushes their data to Google's BigQuery tool, which this gets its data from.
  - briandoll 14y ago
    Correction: We (GitHub) do not push data to BigQuery. Ilya's project archives the GitHub public timeline. That data is then uploaded from his archive into BigQuery. The BigQuery project is public, so anyone can use it.
    - PanMan 14y ago
      Is there any docs on the github public timeline? I did found http://github.com/timeline http://github.com/timeline which made me guess for http://github.com/timeline.json http://github.com/timeline.json which works, but is this documented anywhere? Are there parameters? How often does it refresh? Thnx!
Cyranix 14y ago
I haven't had to deal with this scenario, thankfully, but I seem to recall dimly that there is some mechanism to rescue GitHub users who accidentally publish passwords and access keys. Does the archive respect this same mechanism? (Or did I dream up the mechanism in the first place?)
- omra 14y ago
  There certainly is a mechanism [0], however it doesn't actually completely remove the commit from the remote GitHub repo, as people with the direct link can still access it. I'll check to see if I can find removed commits using this. [0] https://help.github.com/articles/remove-sensitive-data https://help.github.com/articles/remove-sensitive-data

38 points by ig1 14y ago

danso 14y ago

Dumb question, but is the code behind the archive open source? I didn't see any link but did see the star button which... Ok...I see the star button serves as a link https://github.com/igrigorik/githubarchive.org/ https://github.com/igrigorik/githubarchive.org/ It's great to have other crawler projects to learn from

briandoll 14y ago
Yep, the repository is public and a big chunk of this public data is accessible on BigQuery for anyone to query and analyze. Ilya Grigorik and myself just gave a talk at Strata[1] on interesting analysis and visualizations that people did with the GitHub Archive as part of the GitHub data challenge[2]. We'll post a link to the talk soon. We announced during the talk that BigQuery will soon (in the next day or two) have an archive of the GitHub public timeline from February 2011 to now. I can't wait to see what people do with it! [1] http://strataconf.com/stratany2012/public/schedule/speaker/25018 http://strataconf.com/stratany2012/public/schedule/speaker/2... [2] https://github.com/blog/1162-github-data-challenge-winners https://github.com/blog/1162-github-data-challenge-winners
PanMan 14y ago
It is a great project. However, it's not based on crawling, as far as I understand: Github pushes their data to Google's BigQuery tool, which this gets its data from.
- briandoll 14y ago
  Correction: We (GitHub) do not push data to BigQuery. Ilya's project archives the GitHub public timeline. That data is then uploaded from his archive into BigQuery. The BigQuery project is public, so anyone can use it.
  - PanMan 14y ago
    Is there any docs on the github public timeline? I did found http://github.com/timeline http://github.com/timeline which made me guess for http://github.com/timeline.json http://github.com/timeline.json which works, but is this documented anywhere? Are there parameters? How often does it refresh? Thnx!

Cyranix 14y ago

I haven't had to deal with this scenario, thankfully, but I seem to recall dimly that there is some mechanism to rescue GitHub users who accidentally publish passwords and access keys. Does the archive respect this same mechanism? (Or did I dream up the mechanism in the first place?)

omra 14y ago
There certainly is a mechanism [0], however it doesn't actually completely remove the commit from the remote GitHub repo, as people with the direct link can still access it. I'll check to see if I can find removed commits using this. [0] https://help.github.com/articles/remove-sensitive-data https://help.github.com/articles/remove-sensitive-data