4 ms·
Oh nice. I've been doing a lot of crawling myself (http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/ http://blog.marc-se
by rb2k_ 15y ago
Oh nice. I've been doing a lot of crawling myself (http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler/ http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blo...) and I'd love to get my hands on this data. I hope they'll segment their data a bit further.
I personally would LOVE to have a simple list of the domainnames themselves without all of the connections and documents.
Also: Why not just use bittorrent to distribute it?
- Aloisius 15y agoI imagine they don't use bittorrent because it is both very large (TBs) and changes frequently. With S3, you could boot up a bunch of Hadoop processes, pull it (without incurring any bandwidth costs I believe), process it and dump out whatever you want.