3 ms·CommonCrawl is not a vlaid comparison. Most robots.txt target CCBot.by 1vuio0pswjnm7 2y agoCommonCrawl is not a vlaid comparison. Most robots.txt target CCBot.