That's what I'm also thinking of. I'm in very early phase of the development of a search engine. I will _not_ index ad serving pages maybe sometimes even whole

That's what I'm also thinking of. I'm in very early phase of the development of a search engine. I will _not_ index ad serving pages maybe sometimes even whole hosts. Adblock rulesets like easylist will help with this. I know about Million Short, but it's not that comprehensive and AFAIR uses Google's index. Also I'm expecting it will result with a significantly smaller index, so maybe I will be able to finance it from my pocket. Small index will enable offline use - download via torrent the whole index and use it offline. With some incremental updates afterwards. However it would still be in the magnitude of tens of gigabytes, on the condition it would really be small enough. The next level for me would be a HTML-only web browser that would offer permanent reader-mode like behavior. But I have too many projects already.