3 ms·
Navigating the WARC file format
- pronoiac 13y agoWait, Common Crawl is part of Automattic? I had no idea!
- teraflop 13y agoIt's not. The example shows them crawling a page from 102jamzorlando.cbslocal.com, which is hosted by Wordpress.com. Apparently Automattic inserts that recruitment ad into the headers of every site they host. (At least, all the ones I've checked so far.)