Some of my primary interests are building websites that can analyse other websites and for that, I need a keyword index and access to the underlying crawl data

Some of my primary interests are building websites that can analyse other websites and for that, I need a keyword index and access to the underlying crawl data (as in, a response that can point me to the exact file offsets that contain the pages). Think [1] but with keywords instead of URLs. [1] http://index.commoncrawl.org/ http://index.commoncrawl.org/