Google scans emails. It isn't real time. Any links I send to a person with a gmail address will eventually be hit by their malware crawlers. This includes li

Google scans emails. It isn't real time. Any links I send to a person with a gmail address will eventually be hit by their malware crawlers. This includes links one would never guess in the lifetime of a universe. I used to only put http simple auth in place, but now I also null route their IP ranges. I'm sure they will eventually flag my domains as malicious for doing so. Same goes for Slack/Discord/Steam bots. I should also add that these bots are in no way associated with robots.txt and site indexing. The crawlers hitting URL's in emails are not indexing a site, they are looking for malicious content. The contents of robots.txt will have zero impact on this behavior and is not queried at all. I suspect the reason people get conflicting behavior is that Google likely has a mechanism in place that maps domains to levels of trust and connectivity. If a site is known to scan itself for malware, there would be no need for Google to duplicate the effort.