5 ms·
That would be great, scraping that site is the entrypoint of my data pipeline!
by danthelion 3y ago
That would be great, scraping that site is the entrypoint of my data pipeline!
- mlinksva 3y agoJust out of curiosity about how people actually access stuff, does scraping mean (as I'd expect I guess) the rendered website, or consuming the data files https://github.com/github/government.github.com/tree/gh-pages/_data https://github.com/github/government.github.com/tree/gh-page... ? Querying wikidata may also be useful https://github.com/github/government.github.com/issues/877 https://github.com/github/government.github.com/issues/877 Added: note about your project at https://github.com/github/government.github.com/issues/1167 https://github.com/github/government.github.com/issues/1167
- danthelion 3y agoI have missed the data file entirely and just extracted all the entities from the rendered website, then enriched all the repositories with some extra info through the API (still ingesting some data for a few repos, hitting the API limit frequently). The goal is to have a deeper level view of public government orgs, including a more granular view of contributors and how they interact with code. Great idea about wikidata! That's another data rabbit-hole to climb down into.