5 ms·
This whole yep.com thing feels like an attempt to get everyone to please stop blocking their bot
by fooey 4y ago
This whole yep.com thing feels like an attempt to get everyone to please stop blocking their bot
- blinding-streak 4y ago100% this right here. Ahrefs and Yep is not some altruistic endeavor.
- moneywoes 4y agoDumb question but would they not rotate bots in a situation like this? Or do they have a specific crawler
- archerx 4y agoThey have am aggressive crawler that I block. Why would I let a company use my resources to gather my data just so they can sell it to others for a premium at my expense?
- ricardo81 4y agoThey use a specific UA which is in the spirit of robots.txt, you're able to identify and allow/disallow access. Trying to masquerade as another agent would be considered bad form, but obviously happens a lot. There's a similar bot, MJ12Bot that powers Majestic's index which is similar to ahrefs. IIRC they have a user agent but their crawling is distributed, it's impossible to verify whether someone with that UA is them or someone else masquerading. Good practice by bot owners is having a UA and known IPs they crawl from which can be verified by DNS and reverse DNS lookups.