3 ms·
Why doesn't this just use the GitHub public dataset available on Google BigQuery to have much more accurate data rather than "scraping GitHub web search results
by alpb 7y ago
Why doesn't this just use the GitHub public dataset available on Google BigQuery to have much more accurate data rather than "scraping GitHub web search results"? https://cloud.google.com/bigquery/public-data/ https://cloud.google.com/bigquery/public-data/
There are a lot of examples of people analyzing public code on GitHub efficiently for patterns and usages with BigQuery and getting pretty accurate data out of it. https://medium.com/google-cloud/analyzing-go-code-with-bigquery-485c70c3b451 https://medium.com/google-cloud/analyzing-go-code-with-bigqu...
If you use GitHub on a daily basis, you are unlucky enough to know that web search sadly can't even find words that exist in your repository.
- mmcniece 7y agoOne limitation of the BigQuery dataset is they only look at repos with a license on them[1], the scraping approach can look at all public repos. [1] https://github.blog/2017-01-19-github-data-ready-for-you-to-explore-with-bigquery/ https://github.blog/2017-01-19-github-data-ready-for-you-to-...