2 ms·
Dived into Common Crawl’s 45 billion URLs, processing 3TB of data using Python's aiohttp and multiprocessing, Kaggle, and MinIO. Resolved domains with Google’s
by ghafouri 2y ago
Dived into Common Crawl’s 45 billion URLs, processing 3TB of data using Python's aiohttp and multiprocessing, Kaggle, and MinIO. Resolved domains with Google’s and Cloudflare’s DNS over HTTPS, uncovering over 465,000 Shopify domains. Discover the code and data insights here!