3 ms·
Ahrefs operates their own crawler, and gets really valuable data from it. Source: My server logs.
by gkbrk 3y ago
Ahrefs operates their own crawler, and gets really valuable data from it.
Source: My server logs.
- nubinetwork 3y agoThey're really bad at purging outdated search results, I often see them trying to GET pages that haven't existed in 5+ years.
- Kye 3y agoIt isn't just a search engine. It's also a link monitor. Someone somewhere is linking those pages. It might not be a current user, but they still crawl it in case the person with those bad links signs up.