4 ms·
Do Ahrefs et al operate their own crawling infrastructure or is there another third party vendor for this?
by marban 3y ago
Do Ahrefs et al operate their own crawling infrastructure or is there another third party vendor for this?
- gkbrk 3y agoAhrefs operates their own crawler, and gets really valuable data from it. Source: My server logs.
- nubinetwork 3y agoThey're really bad at purging outdated search results, I often see them trying to GET pages that haven't existed in 5+ years.
- Kye 3y agoIt isn't just a search engine. It's also a link monitor. Someone somewhere is linking those pages. It might not be a current user, but they still crawl it in case the person with those bad links signs up.
- skilled 3y agoThey do. And they have their own search engine, too - Yep.com They have some info on their Data[0] page. I always saw them as - if Google Search had a public back-end UI, then Ahrefs would be it. [0]: https://ahrefs.com/big-data https://ahrefs.com/big-data
- smarx007 3y agoThey do: https://ocaml.org/success-stories/peta-byte-scale-web-crawler https://ocaml.org/success-stories/peta-byte-scale-web-crawle...