> Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be t

> Crucially, I would not have it crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers. I like this idea. It would be interesting to see the domains of every search query that I have clicked on and see what the distributions is like. I suspect there would be a long tail but I wonder how many domains actually need to be indexed for 99% of my personal search needs. Does anyone have data like this?