3 ms·
If the mining are done at Google Cloud Storage, initiated by a google search bot, can't Google then identify and handle such abuse? I assume Google already scan
by belorn 12y ago
If the mining are done at Google Cloud Storage, initiated by a google search bot, can't Google then identify and handle such abuse? I assume Google already scans for multiple types of abuse, such as sites that spread malware.
- skj 12y agoNow you're suggesting that Google basically devotes a team to detecting "crawler-free-quota abuse", when the real solution needs to handles crawlers from many different sources that aren't all Google.
- nemothekid 12y agoGoogle shouldn't really have to do this. Replace GoogleBot with BingBot or GCE with AWS and you still have the same problem. A website operator should be working to make sure search crawlers don't consume too many resources given that the bots follow rules. Otherwise you'd have a team at every cloud provider trying to figure out how to manage bots.