3 ms·
In their robots.txt, no crawl delays are given, someone seems to have overlooked that ?__a=2 works just as well as ?__a=1, and overall, whether they intended t
by 1vuio0pswjnm7 6y ago
In their robots.txt, no crawl delays are given, someone seems to have overlooked that ?__a=2 works just as well as ?__a=1, and overall, whether they intended to or not, they appear to be saying they only want to allow certain bots. Perhaps try changing your user-agent string to match the ones in robots.txt.
Generally I do not use middleman sites like Imginn as I find I can perform for myself whatever "service", e.g., downloading resources from some other website, they are offering.