While this won't work too well when the access is indirect via a piracy or a "rogue contractor", it can be applicable to the web-crawlers the companies are dire

While this won't work too well when the access is indirect via a piracy or a "rogue contractor", it can be applicable to the web-crawlers the companies are directly running.