4 ms·
> The code uses browser automation because creating a RegEx pattern for all weird-as-heck "company" names is hard. That doesn't explain why web scraping is use
by minimaxir 10y ago
> The code uses browser automation because creating a RegEx pattern for all weird-as-heck "company" names is hard.
That doesn't explain why web scraping is used instead of using the GitHub API, which is more kosher.
With the API, query all Stargazers, then query the user profiles of those users.
For unauthenticated requests, this will only work if n < 60, but for authenticated requests, the sky is the limit.
I have an old GitHub repo that essentially implements this and I believe it still works: https://github.com/minimaxir/get-profile-data-of-repo-stargazers https://github.com/minimaxir/get-profile-data-of-repo-starga...
It only needs a tweak to support the company field, and I might do just that if there is enough demand.
- minxomat 10y agoWell, I'd like to see what the banner for other projects look like without auth-ing the API (and 60 requests seems a bit low for that). I really just hacked this together on a train without further thought put into it. Sorry again for that.
- sidi 10y agoI believe it's 5000 reqs/hr with an access token.
- andrewmcwatters 10y agoKind of unfortunate, but it works. There's always room for improvement. Good eye, that's a fairly big implementation detail.