I had no idea that googlebot (and others) will sometimes POST to urls[0] - this seems like a terrible idea. I imagine what happened to this guy is that somebod

I had no idea that googlebot (and others) will sometimes POST to urls[0] - this seems like a terrible idea. I imagine what happened to this guy is that somebody coded up a site that used his as a service and google spidered it and went crazy trying to follow the links. It sucks, but that is what robots.txt is for. Anything that requires more than just serving a page should be blocked. You can drive yourself insane reading your logs trying to guess what bots are trying to achieve. They seem to excel at spidering exactly what you don't want while ignoring the content that would actually be useful. The one time I did have a problem with a misbehaving crawler was with something called 80legs[1] [0] https://webmasters.googleblog.com/2011/11/get-post-and-safely-surfacing-more-of.html https://webmasters.googleblog.com/2011/11/get-post-and-safel... [1] https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html