4 ms·

I think the biggest challenge for a search engine is getting the corpus to search — the crawling and scraping. A collective of search engines could go in toget

I think the biggest challenge for a search engine is getting the corpus to search — the crawling and scraping. A collective of search engines could go in together on that together. Then they could go their own way on indexing it and making it useful.

dredmorbius 5y ago
One possible solution to this would be for sites to offer their own indexes. If you think about it, search is an emergent sector that developed because there was no standard for sites to index themselves. Compiling that index and providing that would be far more efficient than trying to crawl billions (or is it trillions) of pages independently by multiple individual search engines, and would remove most of the incumbent advantage of such platforms. Instead we ended up with a few pieces of this --- robots.txt and sitemaps --- but not a full solution. Yes, you'd need to check for cheats (sample pages based on the index and look for signs of stuffing or misrepresentation, penalise accordingly). Reputation systems were based on inbound and outbound links, so those might also be offered. I'm not entirely sure this would work, but I'd be interested in seeing the concept fleshed out further.
- onionisafruit 5y ago
  That’s an interesting point about search developing in an absence of a standard for sites to self-index. I can’t quite imagine what that standard index would look like.
  - dredmorbius 5y ago
    It would probably look much like the search index that search engines use. These consist of both individual terms and tuples. (Phrase search can be constructed reasonably reliably by chaining ngram tuples.) Rather than transferring the entire website, it would only be necessary to transfer the index itself. Mind that a sufficiently-large ngram-tuple index is itself large, I suspect limits to 3-word ngrams might be necessary, and should be reasonably sufficient. I'm kicking around the prospect of a DNS-like query mechanism, with cache-and-forward capabilities. That could both distribute search and indexing load. Reputation assessment and management remains the hard piece. Early computer science focused on the relatively easy problem of string search. At Web scale, the problem is of idnetifying reliable and useful sources. That itself isn't a static problem with a single solution as the landscape evolves in response to methods.
leobg 5y ago
Hm. Isn’t the web itself the corpus? A collective scrape would be as big as the web itself. (In order to “go their own way on indexing” they’d all need the full ground truth pages.)

dredmorbius 5y ago

One possible solution to this would be for sites to offer their own indexes. If you think about it, search is an emergent sector that developed because there was no standard for sites to index themselves. Compiling that index and providing that would be far more efficient than trying to crawl billions (or is it trillions) of pages independently by multiple individual search engines, and would remove most of the incumbent advantage of such platforms. Instead we ended up with a few pieces of this --- robots.txt and sitemaps --- but not a full solution. Yes, you'd need to check for cheats (sample pages based on the index and look for signs of stuffing or misrepresentation, penalise accordingly). Reputation systems were based on inbound and outbound links, so those might also be offered. I'm not entirely sure this would work, but I'd be interested in seeing the concept fleshed out further.

onionisafruit 5y ago
That’s an interesting point about search developing in an absence of a standard for sites to self-index. I can’t quite imagine what that standard index would look like.
- dredmorbius 5y ago
  It would probably look much like the search index that search engines use. These consist of both individual terms and tuples. (Phrase search can be constructed reasonably reliably by chaining ngram tuples.) Rather than transferring the entire website, it would only be necessary to transfer the index itself. Mind that a sufficiently-large ngram-tuple index is itself large, I suspect limits to 3-word ngrams might be necessary, and should be reasonably sufficient. I'm kicking around the prospect of a DNS-like query mechanism, with cache-and-forward capabilities. That could both distribute search and indexing load. Reputation assessment and management remains the hard piece. Early computer science focused on the relatively easy problem of string search. At Web scale, the problem is of idnetifying reliable and useful sources. That itself isn't a static problem with a single solution as the landscape evolves in response to methods.

leobg 5y ago

Hm. Isn’t the web itself the corpus? A collective scrape would be as big as the web itself. (In order to “go their own way on indexing” they’d all need the full ground truth pages.)