3 ms·
Common Crawl maintains a free, open repository of web crawl data
- deleted 2y ago[deleted]
- tony-allan 2y agoI haven't seen this resource before. You can also search the index [1] I downloaded and ran the example code [2] to lookup a URL and fetch its content and the response was instantaneous! [1] http://index.commoncrawl.org/CC-MAIN-2025-08-index?url=ycombinator.com&output=json [2] https://commoncrawl.org/get-started