40 ms·
> you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events https://api.github.com/events W
by KirillPanov 4y ago
> you can subscribe to a stream of ALL events happening at GitHub by fetching from this endpoint: https://api.github.com/events https://api.github.com/events
Wow I am shocked that they allow "firehose" access not only for free, but without even an API key.
Given enough disk and bandwidth, does this mean you could keep your own copy of all of github? I'd love to be able to grep the whole thing.
- jonas-w 4y agohttps://www.gharchive.org/ https://www.gharchive.org/
- bspammer 4y agoThe leaked private key should be somewhere in this archive then no?
- jonas-w 4y agoThe events stream does not contain files or anything in your git repo. Only stuff like commit messages, issues, comments, etc.
- 112 4y agoi think it does contain code
- squeaky-clean 4y agoYou can view it yourself, it doesn't contain the code directly, but it does contain a link to the code blob.
- KirillPanov 4y agoWow thanks. I looked all over that page but couldn't get a rough idea of the dataset size for the event stream (not the commits). Are we talking 1TB/year, 10TB/year, 100TB/year?
- jonas-w 4y agoYour comment made me curious. I downloaded the complete last month (February). 28 days, every day has a gzipped json file which is on average 86MB. Summarized it is 53GB of gzipped jsons for 28 days. Roughly 700GB gzipped per year. Didn't have the chance to ungzip them all.