4 ms·
If by offender you mean the bot, then I'm confused. The bot asked for robots.txt in a plaintext format. The bot was delivered some binary garbage that it couldn
by gnfargbl 4y ago
If by offender you mean the bot, then I'm confused. The bot asked for robots.txt in a plaintext format. The bot was delivered some binary garbage that it couldn't parse. The bot continued to ask for an updated robots.txt, and continued to be told that the binary garbage was the intended content. What more was it supposed to do, exactly? The offending party here is the broken hosting platform.
- xypage 4y agoI don't think banning the offender is in order to punish them for doing something wrong so much as to manually ensure it's not crawling your site