3 ms·
It's common to see them get stuck in a loop on online stores trying every combination of product filter over and over. Even Googlebot has to be told to not cra
by dwd 1y ago
It's common to see them get stuck in a loop on online stores trying every combination of product filter over and over.
Even Googlebot has to be told to not crawl particular querystrings, but the AI crawlers are worse.
- rco8786 1y agoNot an online store just a bog standard Wordpress site