7 ms·
It's weird that Google is charging for the GoogleBot bandwidth on its own services. Of course they absolutely don't have to do it, but stories like this make me
by arcatek 12y ago
It's weird that Google is charging for the GoogleBot bandwidth on its own services. Of course they absolutely don't have to do it, but stories like this make me worry about using the Google Cloud Storage.
[edit] That being said, the issue would be the same if it was another hoster or another search engine. I guess the real solution would be to be able to limit the crawl rate, as the OP said.
- thekeywordgeek 12y agoAbsolutely, It's only coincidence that Google are both the host and the spider. I really would appreciate the ability to throttle it though.
- skj 12y agoI thought about that too (I'm a Googler working on cloud), but then a colleague mentioned that this would become a way to get free computation from Google. So, while I agree with the sentiment that it sucks that this crawling eats the quota, the solution is not to simply bypass the quota.
- cordite 12y agoWhat about discounted? It is rather unfair for Google to be eating both the funds and the service here. That, and adding proper support for tuning the crawl rate.
- thekeywordgeek 12y agoAbsolutely, even I can't say I'm deserving of a free lunch.
- DanBC 12y ago> but then a colleague mentioned that this would become a way to get free computation from Google I'm a bit confused. What computation does the GoogleBot cause to be performed that benefits the Google service user? (Not Googlebot related stuff like indexing). EDIT: Thanks kyrra!
- kyrra 12y agoHave a bunch of pages with no real content (but have millions of pages). Everytime someone tries to load a page, do some intensive task (ex: mining bitcoins). If you just make it appealing to GoogleBot and no one else, you get free computational resources.
- xigency 12y agoIs that really how it works?
- belorn 12y agoIf the mining are done at Google Cloud Storage, initiated by a google search bot, can't Google then identify and handle such abuse? I assume Google already scans for multiple types of abuse, such as sites that spread malware.
- skj 12y agoNow you're suggesting that Google basically devotes a team to detecting "crawler-free-quota abuse", when the real solution needs to handles crawlers from many different sources that aren't all Google.
- nemothekid 12y agoGoogle shouldn't really have to do this. Replace GoogleBot with BingBot or GCE with AWS and you still have the same problem. A website operator should be working to make sure search crawlers don't consume too many resources given that the bots follow rules. Otherwise you'd have a team at every cloud provider trying to figure out how to manage bots.
- fragmede 12y agoSorry, I still don't get it. How does not charging for outgoing network traffic make computation free? You'd still be paying for everything else, eg the instances themselves, datastore storage, read/write datastore calls, using the logs API, which means mining bitcoins wouldn't be free.
- deleted 12y ago[deleted]
- Someone1234 12y agoWhen your cloud provider lets a client run their own source code, how can you REALLY determine that incoming traffic is even from a crawler? Do you want them to spawn a specific instance of the app just used by Googlebot and then using a load balancer to redirect those requests to those specific instances? The more you think about this, the more insanely complex it gets.
- skj 12y agoGoogle crawlers come from well-known IPs, especially well-known to Google. Appengine requests come through reverse proxies, and there is no fundamental difficulty in not counting requests from crawlers towards the quota. That said, see my other descendant comment.