You should look for sitemaps. They are often listed in robots.txt. Will save you lots of time! These sites have sitemaps: http://torrentproject.se/robots.txt h

You should look for sitemaps. They are often listed in robots.txt. Will save you lots of time! These sites have sitemaps: http://torrentproject.se/robots.txt http://torrentproject.se/robots.txt https://www.torrentdownloads.me/robots.txt https://www.torrentdownloads.me/robots.txt https://thepiratebay.org/robots.txt https://thepiratebay.org/robots.txt http://goldtorrents.com/robots.txt http://goldtorrents.com/robots.txt https://bitsnoop.com/robots.txt https://bitsnoop.com/robots.txt Enough with the hype: "high performance" What?? Your program is not concurrent: you have one process per website and go through URLs one at a time. You don't even use Bloom filters.