6 ms·
Ask HN: How do companies like Moz get search data without getting banned?
I know that Google bans scrapers outright and anyone trying to shadily get around the system gets caught at one point or another. So how is it that large companies like Moz, Ahrefs, and others are able to get all the data they need, quickly, without getting banned, and I swear that I've seen Matt Cutts even link back to Ahrefs in the past.
- moron4hire 13y agoI would also like to know.
- jotato 13y agoI worked for a competitor to Moz. We used a service like Anonymizer to hide it. My guess is they do as well.
- caruana 13y agoThey probably pay for it. Or rather google gives them a paid account but doesn't charge them. https://support.google.com/customsearch/answer/72334?hl=en https://support.google.com/customsearch/answer/72334?hl=en
- dangrossman 13y agoThat's pricing for Google Site Search, a Google-hosted search engine of your own site. It's not a way to buy SERPs for the entire Google web index which is what SEO services need.
- cynusx 13y agoAhrefs crawls the internet just like Google does. SeoMoz does the same: http://moz.com/help/pro/what-is-rogerbot- http://moz.com/help/pro/what-is-rogerbot- Every service that builds a backlink database uses their own crawlers So to answer the question on how they get their data so fast, they OWN the data and they use big hadoop farms to crunch it and prepare it for api retrieval.
- staunch 13y agoIt's not hard to setup a bunch of machines on various networks with hundreds of IPs. You can't really block people with resources.