You may not need a crawler for this. The HTTP Archive has already done all the crawling for a million URLs from the most popular websites. If one of their exi

You may not need a crawler for this. The HTTP Archive has already done all the crawling for a million URLs from the most popular websites. If one of their existing reports doesn't cover what you need, you can write custom queries using BigQuery: https://httparchive.org/faq#how-do-i-use-bigquery-to-write-custom-queries-over-the-data https://httparchive.org/faq#how-do-i-use-bigquery-to-write-c...