One possible issue is that the users of the data may place different prominence on a particular URL, and may decide that it warrants more frequent crawling. Fo

One possible issue is that the users of the data may place different prominence on a particular URL, and may decide that it warrants more frequent crawling. For site owners, it may prove to be lower maintenance to simply whitelist such a bot for crawling, though on the downside they may want to pick and choose which search engines can index it, and since the crawl (assumedly) would use one UA it would complicate how site owners can tell respective search engines what they're allowed to do with the crawl data.