> I'd say not holding to the standards of robots.txt and 403-Forbidden is quite malicious Most hidden services don't publish robots files. The only ones that d

> I'd say not holding to the standards of robots.txt and 403-Forbidden is quite malicious Most hidden services don't publish robots files. The only ones that do are the proxy services (which are hidden services but not usually 'hidden'). The purpose of the proxying is to find, discover and monitor what are usually illegal or malicious services. I don't think there are legitimate crawlers on hidden services - there are a couple of drug market search engines but they identify themselves outside of robots.txt It's really difficult to run a large-scale hidden service because of this - you need to be able to throttle or block connections but not based on the inbound circuit. You also need to setup guards (which OP makes no mention of) > It overloads the network, doesn't crawl and doesn't parse the responses. It's likely adding those later responses into a crawl queue that is tens of thousands of URLs long. Overloading the network is unintentional, usually your crawling is throttled by your circuit.