3 ms·
Cycling through a couple of user agent strings to pretend to be a real user takes only few lines of code. Are images requested and Javascript executed? I would
by mtmail 11y ago
Cycling through a couple of user agent strings to pretend to be a real user takes only few lines of code. Are images requested and Javascript executed?
I wouldn't read too much into this. There are lots of crawler/spiders around. From product search engines, people search engines, image or audio, generic or research, some crawlers' whole purpose is to point to your referrers (ads) or maybe your competition hired somebody to crawl you.
Once you have a website for a while those crawlers are as usual as programs that try to login or look for old Wordpress vulnerabilities (/wp-admin, /cgi-bin/ etc).