Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index. W

Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index. What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.