3 ms·
I thought about that too (I'm a Googler working on cloud), but then a colleague mentioned that this would become a way to get free computation from Google. So,
by skj 12y ago
I thought about that too (I'm a Googler working on cloud), but then a colleague mentioned that this would become a way to get free computation from Google.
So, while I agree with the sentiment that it sucks that this crawling eats the quota, the solution is not to simply bypass the quota.
- cordite 12y agoWhat about discounted? It is rather unfair for Google to be eating both the funds and the service here. That, and adding proper support for tuning the crawl rate.
- thekeywordgeek 12y agoAbsolutely, even I can't say I'm deserving of a free lunch.
- DanBC 12y ago> but then a colleague mentioned that this would become a way to get free computation from Google I'm a bit confused. What computation does the GoogleBot cause to be performed that benefits the Google service user? (Not Googlebot related stuff like indexing). EDIT: Thanks kyrra!
- kyrra 12y agoHave a bunch of pages with no real content (but have millions of pages). Everytime someone tries to load a page, do some intensive task (ex: mining bitcoins). If you just make it appealing to GoogleBot and no one else, you get free computational resources.
- xigency 12y agoIs that really how it works?
- belorn 12y agoIf the mining are done at Google Cloud Storage, initiated by a google search bot, can't Google then identify and handle such abuse? I assume Google already scans for multiple types of abuse, such as sites that spread malware.
- skj 12y agoNow you're suggesting that Google basically devotes a team to detecting "crawler-free-quota abuse", when the real solution needs to handles crawlers from many different sources that aren't all Google.
- nemothekid 12y agoGoogle shouldn't really have to do this. Replace GoogleBot with BingBot or GCE with AWS and you still have the same problem. A website operator should be working to make sure search crawlers don't consume too many resources given that the bots follow rules. Otherwise you'd have a team at every cloud provider trying to figure out how to manage bots.
- fragmede 12y agoSorry, I still don't get it. How does not charging for outgoing network traffic make computation free? You'd still be paying for everything else, eg the instances themselves, datastore storage, read/write datastore calls, using the logs API, which means mining bitcoins wouldn't be free.
- brianpgordon 12y agoOP's concern isn't with network traffic, it's with GAE compute time. Googlebot keeps causing instances to run. If requests initiated by Googlebot were free to run, you could make a giant website full of garbage and use each free request to spend 50ms mining bitcoin.
- deleted 12y ago[deleted]