4 ms·
No, any web crawler would have done the same thing. It is simply an error to modify content with a GET.
by Tichy 19y ago
No, any web crawler would have done the same thing. It is simply an error to modify content with a GET.
- TFrancis 19y agoAdditionally, the interface to web crawlers (robots.txt) is well defined.
- meat-eater 19y agoI agree. Google simply followed the the semantics of http. They had similar problems with the the Google Web Accelerator before. But it's really hard to blame google for the mistake of other programmers.
- sdurkin 19y agoYes, it is an error. What alarms me is Google's attitude. What happens when we're dealing with interfaces that are a little bit more ill-defined? Will Google continue to demand that you follow their way of doing things? Google's attitude in this case suggests they will.
- neilc 19y agoIt's not "their way of doing things", it is just the way the web works (per HTTP spec). Any crawler would have done the same thing in that situation -- that fact that it happened to be Google is merely coincidental. Given the scale at which they operate, you can't expect Google or any other web-scale crawler to be mind-readers.