4 ms·
Are you willing to post your code for your crawler some where? Sounds like a neat project, shame most people aren't fixing the problem.
by azemetre 8y ago
Are you willing to post your code for your crawler some where? Sounds like a neat project, shame most people aren't fixing the problem.
- JimDabell 8y agoYou may not need a crawler for this. The HTTP Archive has already done all the crawling for a million URLs from the most popular websites. If one of their existing reports doesn't cover what you need, you can write custom queries using BigQuery: https://httparchive.org/faq#how-do-i-use-bigquery-to-write-custom-queries-over-the-data https://httparchive.org/faq#how-do-i-use-bigquery-to-write-c...
- andrethegiant 8y agoMaybe one day. Right now it's really messy. It uses crawler[0] and rethinkdb[1] FYI. [0] https://www.npmjs.com/package/crawler https://www.npmjs.com/package/crawler [1] https://rethinkdb.com https://rethinkdb.com