3 ms·

> ahrefs or semrush OK I gotta ask: these 2 particular bots constantly scan my site once an hour. Why?

• throwaway77384 7y ago
  Complete speculation: They make their money by assessing back-links. So they crawl as many websites as possible, as often as possible, to establish some kind of view of who is linking to what. From that, they extrapolate some kind of popularity score, which they can probably correlate with keywords and types of users and searches. That is what they sell back to users, presumably. I am actually quite interested to see how clear their data / recommendations would be, but haven't had the time to do so yet.
  • Cenk 7y ago
    > I am actually quite interested to see how clear their data / recommendations would be, but haven't had the time to do so yet. I have tried both, and while ahrefs probably has a better dataset, semrush does very specific recommendations when it comes to link building for example. They will suggest what sites to contact to ask for backlinks (something I have not yet tried, because other people email me all the time asking for backlinks or guest posts so I know how annoying it is).
• andygcook 7y ago
  It's likely because other companies/people are tracking your site as a competitor in Ahrefs/SEMrush.
• alvah 7y ago
  There's an arms race in the SEO tools market, to have the largest / most frequently updated / most accurate backlinks database. Ahrefs have used the claim that their bot is the most active (apart from Googlebot) in their marketing.
• Cenk 7y ago
  Basically what the other commenters said, they scan the entire web to find out who links to whom and how many backlinks each site has. One thing you can do is stop them from scanning your site with robots.txt – some SEO people recommend this to prevent your competitors from finding out where you got your backlinks from, but I haven’t tried it.