4 ms·
CommonCrawl: an open repository of web crawl data that is universally accessible
- abhishektwr 15y agoJust a pointer, the code for CommonCrawl Project is available on Github https://github.com/commoncrawl/commoncrawl https://github.com/commoncrawl/commoncrawl
- nithinag 15y agoThis looks really nice!
- Aloisius 15y agoI hear a lot of people are crunching on CommonCrawl's data. It'll be interesting the type of stuff people come up with!
- fungi 15y agoIf you into said things then maybe http://yacy.net/ http://yacy.net/ (p2p crawler and search) will be useful to you as well.
- pooyak 15y agothread on HN from when common crawl was announced, interesting info there: http://news.ycombinator.com/item?id=3209690 http://news.ycombinator.com/item?id=3209690
- rgrieselhuber 15y agoIt would be great to hear more about the tools they are using to crawl and potentially open it up to more people who want to contribute computing resources.
- deleted 15y ago[deleted]
- Titanous 15y agoThe latest data available is from 2010-09-25, which seems to be too old to be useful for most things.
- emilis_info 15y agoThis one may be also interesting for open data devs: http://scraperwiki.com/ http://scraperwiki.com/