3 ms·
robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (
by yongjik 6y ago
robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt!
Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that generated GET requests - because they are supposed to be safe.
Don't be like that site owner.
- YetAnotherNick 6y agoHow do I use robots.txt to tell google to not add item to the shopping cart?
- yongjik 6y agoErm... hide the shopping cart page behind robots.txt?
- kabacha 6y agoAs someone who has seen way too many robots.txt files that's exactly how you do it.
- rubyron 6y agoWell, theoretically, your Add To Cart button could have an href with a path that’s banned in robots.txt, but overridden with JS. But most online stores should be happy to have Google crawling their prices and showing up under the Shopping results.
- a1369209993 6y agoThat has nothing to do with robots.txt, the problem is doing things in response to GET requests. I've said it before and I'll say it again: you do not do things on GET requests.
- deleted 6y ago[deleted]
- ehsankia 6y agoI'm curious though, why didn't Google properly tag their robot in this case? Or was the reporter did not know about user agents? It seems strange for them to crawl with a "mystery bot".
- eythian 6y agoI think you're thinking of this: https://thedailywtf.com/articles/The_Spider_of_Doom https://thedailywtf.com/articles/The_Spider_of_Doom - which obviously has two issues: the auth, and the actions on GET.