3 ms·

I created a dashboard to monitor government activity on GitHub

taubek 3y ago
What is the source of your data? How do you define "government activity on GitHub"? Do you look at domain used in contact, affiliation, etc.?
- danthelion 3y ago
  This GitHub page was the starting point: https://government.github.com/community https://government.github.com/community
  - Dunstark 3y ago
    From what I gather from this page it seems organisations need to be added by the community or organisation members. I know a few organisations that aren't on that list for my country like https://github.com/GouvernementFR https://github.com/GouvernementFR, https://github.com/DISIC https://github.com/DISIC, https://github.com/ANSSI-FR https://github.com/ANSSI-FR and https://github.com/ansforge https://github.com/ansforge. No promises but I might make a PR tonight on the page you mentioned.
    - danthelion 3y ago
      That would be great, scraping that site is the entrypoint of my data pipeline!
      - mlinksva 3y ago
        Just out of curiosity about how people actually access stuff, does scraping mean (as I'd expect I guess) the rendered website, or consuming the data files https://github.com/github/government.github.com/tree/gh-pages/_data https://github.com/github/government.github.com/tree/gh-page... ? Querying wikidata may also be useful https://github.com/github/government.github.com/issues/877 https://github.com/github/government.github.com/issues/877 Added: note about your project at https://github.com/github/government.github.com/issues/1167 https://github.com/github/government.github.com/issues/1167
        danthelion 3y ago
        I have missed the data file entirely and just extracted all the entities from the rendered website, then enriched all the repositories with some extra info through the API (still ingesting some data for a few repos, hitting the API limit frequently). The goal is to have a deeper level view of public government orgs, including a more granular view of contributors and how they interact with code. Great idea about wikidata! That's another data rabbit-hole to climb down into.
  - l__l 3y ago
    Even from this page there appears to be a fair bit missing. E.g. one of the biggest UK gov orgs is @alphagov (~1.6k repos), which I can't seem to see on your dash?
  - rickette 3y ago
    For The Netherlands only a couple of organizations are listed there, lots are missing.
stuaxo 3y ago
What's the aim ?

What is the source of your data? How do you define "government activity on GitHub"? Do you look at domain used in contact, affiliation, etc.?

danthelion 3y ago
This GitHub page was the starting point: https://government.github.com/community https://government.github.com/community
- Dunstark 3y ago
  From what I gather from this page it seems organisations need to be added by the community or organisation members. I know a few organisations that aren't on that list for my country like https://github.com/GouvernementFR https://github.com/GouvernementFR, https://github.com/DISIC https://github.com/DISIC, https://github.com/ANSSI-FR https://github.com/ANSSI-FR and https://github.com/ansforge https://github.com/ansforge. No promises but I might make a PR tonight on the page you mentioned.
  - danthelion 3y ago
    That would be great, scraping that site is the entrypoint of my data pipeline!
    - mlinksva 3y ago
      Just out of curiosity about how people actually access stuff, does scraping mean (as I'd expect I guess) the rendered website, or consuming the data files https://github.com/github/government.github.com/tree/gh-pages/_data https://github.com/github/government.github.com/tree/gh-page... ? Querying wikidata may also be useful https://github.com/github/government.github.com/issues/877 https://github.com/github/government.github.com/issues/877 Added: note about your project at https://github.com/github/government.github.com/issues/1167 https://github.com/github/government.github.com/issues/1167
      - danthelion 3y ago
        I have missed the data file entirely and just extracted all the entities from the rendered website, then enriched all the repositories with some extra info through the API (still ingesting some data for a few repos, hitting the API limit frequently). The goal is to have a deeper level view of public government orgs, including a more granular view of contributors and how they interact with code. Great idea about wikidata! That's another data rabbit-hole to climb down into.
- l__l 3y ago
  Even from this page there appears to be a fair bit missing. E.g. one of the biggest UK gov orgs is @alphagov (~1.6k repos), which I can't seem to see on your dash?
- rickette 3y ago
  For The Netherlands only a couple of organizations are listed there, lots are missing.

stuaxo 3y ago

What's the aim ?