> web masters to become hostile to crawlers You must have been living under a rock for the past decade, haven't you? Web masters are definitely hostile towards

> web masters to become hostile to crawlers You must have been living under a rock for the past decade, haven't you? Web masters are definitely hostile towards webcrawlers. There's an entire platitude of "web-crawler" protection services - cloudflare for example is probably the biggest one.