4 ms·
Microsoft's bot used to do that kind of thing - not sure if it's been fixed yet. AFAIK, it's just stupid and there's no way to fix it. You can set crawl-delay
by JoachimSchipper 16y ago
Microsoft's bot used to do that kind of thing - not sure if it's been fixed yet. AFAIK, it's just stupid and there's no way to fix it.
You can set crawl-delay in robots.txt if you want to reduce load, but you don't appear to need it. Note that, again, MS' bot doesn't honour this.
It may be sensible to make the tag pages noindex, and a sitemap.xml may help these bots. But I have no other ideas.