3 ms·
We did exactly this at SEOmoz - built an index that's primarily focused on link data. It's accessible through a free API - www.seomoz.org/api - up to 1 million
by randfish 17y ago
We did exactly this at SEOmoz - built an index that's primarily focused on link data. It's accessible through a free API - www.seomoz.org/api - up to 1 million calls a month free and more at fairly low rates. Our index isn't quite Google-sized. Our last crawl (updated once per month) was 65 billion URLs, which we estimate is between 50-65% of Google's main index.
I'm happy to answer any other questions you've got about it if you'd like. Feel free to email me (rand at seomoz dot org).