4 ms·
Web Crawl Analysis: Unveiling 465K Shopify Domains from 3TB of Data
- ghafouri 2y agoDived into Common Crawl’s 45 billion URLs, processing 3TB of data using Python's aiohttp and multiprocessing, Kaggle, and MinIO. Resolved domains with Google’s and Cloudflare’s DNS over HTTPS, uncovering over 465,000 Shopify domains. Discover the code and data insights here!
- SV_BubbleTime 2y agoI don’t get it. Why is scraping Shopify domains a goal / interesting?
- tommek4077 2y agoMaybe to use known vulnerabilities and take over sites?
- leobg 2y agoCame here to ask the same thing. Also, if they are all on the same IP, what about reverse dns lookups?
- ghafouri 2y agoWith reverse DNS, I only get a few results, not hundreds of thousands of domains. So, I took this approach.
- xnx 2y agoGenerate a list of leads for Shopify related services or Shopify competing product.
- ghafouri 2y agoI used to work on a Shopify SaaS project, but it was canceled. I'm thinking of sharing some of my work from that project.
- neilv 2y agoWhy pound Web services of Google or CloudFlare for DNS lookups, instead of using the DNS protocol? Were DNS servers balking at the continuous 100 simultaneous requests to resolve much of the Internet?
- ghafouri 2y agoI achieved 1000 DNS resolves per second with DOH, and my attempts on DNS servers are fewer.
- yodon 2y agoHave you thought about productizing this as a better reverse DNS lookup? It feels like you're 90% of the way there.