4 ms·

P2P is probably the best solution for search in the long run, but here is how it can be decentralized in the interim. Separate crawlers, indexers and search we

P2P is probably the best solution for search in the long run, but here is how it can be decentralized in the interim. Separate crawlers, indexers and search websites. One search website could go to several indexers. One indexer could get info from several crawlers. In today's internet this makes so much more sense than everything being run by the same company. For example, you could have a crawler that specializes at searching Medium.com. You could have another one that specializes in searching just computer game websites. The data from both are aggregated by indexer A and indexer B. Indexers would rank content using different algorithms. When user opens some search website and types a query, the website pre-processes the query and sends it to both indexers. Both return the best results they can. The website then aggregates them into a single page and shows to the user. With this system there will be competition at every level. And a startup could conceivably build a node in this infrastructure without tackling the entire thing at the same time (like you have to do right now). You think website X has bad interface? You can build your own interface and buy information from existing indexers (which you chose). You think you can do better job at ranking results? You can build your own indexer and sell your results to pre-existing websites. You think some niche area of the web is ignored? You can build your own crawler and sell your results to one or several indexers. If someone does a bad job, the next level of the system has incentives (and freedom) to drop them. This includes the consumer of search results.

kleer001 7y ago
Now THAT sounds like a fertile ecosystem.
MadWombat 7y ago
> In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on HN, so I agree with you that this would be a really neat to have an ecosystem like this. But the question is how do you make money? What is your incentive to run a crawler? What is your incentive to run an indexer? People are used to the idea that web search is free, so you search sites cannot charge people money. It also means that they cannot buy indexer services.
- iovrthoughtthis 7y ago
  What about open, distributed indexes instead of companies? Sites that wished to appear in searches would have an incentive to get added to an index. That could create a market for compute to calculate and update indexes.
- dcolkitt 7y ago
  I think it could work with micro transactions on the backend. In B2C Internet consumers won't pay for content. But in B2B its the norm. Paying for an indexer or crawler is really no different than the umpteen million AWS transactions that occur everyday. Most consumers probably won't want to pay for search. At least initially until the culture changes. But that's fine, they don't have to access the crawler or indexers directly. Instead they can point their browser at the frontend URL of their choice. And that frontend can monetize by injecting ads or whatever other business model they want. As long as they earn more in ads than they pay on the open market for backend crawlers, indexers, etc., then they're a viable business. Given how ridiculously profitable Google is, this shouldn't be a difficult hurdle to clear.
  - MadWombat 7y ago
    The requirements in infrastructure are not symmetrical here. An indexer requires a significantly bigger investment in both storage and computational power than either a crawler or an end-user site. Seems like everyone would want to run a crawler, but nobody would want to run an indexer. Also, it is unclear how to start something like this. A sort of a chicken and egg problem where you cannot start running any of these components because there is nobody running the other parts.
    - drmabus 7y ago
      the egg came first. on a serious note, yacy is good enough to crawl a bunch of sites you like and get much superior search results be it limited to your own interest. google is complete garbage. specially nowadays google just returns unrelated cnn pages. try search for corona virus map, I get tons of news articles talking about the same page but the page is not in the results. whatever you add to the query it keeps returning the same news articles from the same sources. if you populate yacy with pages about the topic you will yield real results.
bobajeff 7y ago
> Separate crawlers, indexers and search websites How would all of those be funded? Also, wouldn't sperate indexes be Zero Sum just like Search Engines are?
6510 7y ago
YaCy does seem to already have topical crawlers, they are just users who crawl topics they find interesting. You not just get more than "the" website but [ideally] every small site they could find.
smoyer 7y ago
yacy has support for additional crawlers. But I was thinking of something more radical ... let the indexes be peer-to-peer with an IPFS storage back-end, but build at least the generic crawler as a browser plugin that also performs searches against the index. For "normal" web pages, each visit results in a reindexing. Of course you have to solve the chicken and egg problem of pages that aren't visited because nobody has indexed them so you don't find them in search.
- iovrthoughtthis 7y ago
  Allow people to submit their sites to the index. People who want to be found have an incentive to do the work.
  - Polylactic_acid 7y ago
    People already do this. You can submit your sitemap to bing and google. But I doubt you will get anyone to bother with a small player or p2p service.
bawolff 7y ago
I feel like there is either 2 possibilities: Small number of indexes (like HotBot before google ruled the internet). I'm not sure an oligopoly is that much better than a monopoly Or you have a lot of indexes. But then the system scales really poorly. All the indexes have to handle a large number of queries and be hugely powerful, and all the clients have to spend a ton of time querrying and aggregating them.
OJFord 7y ago
Yes! And of course a big 'enough' publisher like Medium would be incentivised to provide its own crawler implementing the standard interface, for SEO.
tobylane 7y ago
The winner will be the vertical integrator, not the best quality. People will pick the quantitative measure (speed) over anything qualitative.

kleer001 7y ago

Now THAT sounds like a fertile ecosystem.

MadWombat 7y ago

> In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on HN, so I agree with you that this would be a really neat to have an ecosystem like this. But the question is how do you make money? What is your incentive to run a crawler? What is your incentive to run an indexer? People are used to the idea that web search is free, so you search sites cannot charge people money. It also means that they cannot buy indexer services.

iovrthoughtthis 7y ago
What about open, distributed indexes instead of companies? Sites that wished to appear in searches would have an incentive to get added to an index. That could create a market for compute to calculate and update indexes.
dcolkitt 7y ago
I think it could work with micro transactions on the backend. In B2C Internet consumers won't pay for content. But in B2B its the norm. Paying for an indexer or crawler is really no different than the umpteen million AWS transactions that occur everyday. Most consumers probably won't want to pay for search. At least initially until the culture changes. But that's fine, they don't have to access the crawler or indexers directly. Instead they can point their browser at the frontend URL of their choice. And that frontend can monetize by injecting ads or whatever other business model they want. As long as they earn more in ads than they pay on the open market for backend crawlers, indexers, etc., then they're a viable business. Given how ridiculously profitable Google is, this shouldn't be a difficult hurdle to clear.
- MadWombat 7y ago
  The requirements in infrastructure are not symmetrical here. An indexer requires a significantly bigger investment in both storage and computational power than either a crawler or an end-user site. Seems like everyone would want to run a crawler, but nobody would want to run an indexer. Also, it is unclear how to start something like this. A sort of a chicken and egg problem where you cannot start running any of these components because there is nobody running the other parts.
  - drmabus 7y ago
    the egg came first. on a serious note, yacy is good enough to crawl a bunch of sites you like and get much superior search results be it limited to your own interest. google is complete garbage. specially nowadays google just returns unrelated cnn pages. try search for corona virus map, I get tons of news articles talking about the same page but the page is not in the results. whatever you add to the query it keeps returning the same news articles from the same sources. if you populate yacy with pages about the topic you will yield real results.

bobajeff 7y ago

> Separate crawlers, indexers and search websites How would all of those be funded? Also, wouldn't sperate indexes be Zero Sum just like Search Engines are?

6510 7y ago

YaCy does seem to already have topical crawlers, they are just users who crawl topics they find interesting. You not just get more than "the" website but [ideally] every small site they could find.

smoyer 7y ago

yacy has support for additional crawlers. But I was thinking of something more radical ... let the indexes be peer-to-peer with an IPFS storage back-end, but build at least the generic crawler as a browser plugin that also performs searches against the index. For "normal" web pages, each visit results in a reindexing. Of course you have to solve the chicken and egg problem of pages that aren't visited because nobody has indexed them so you don't find them in search.

iovrthoughtthis 7y ago
Allow people to submit their sites to the index. People who want to be found have an incentive to do the work.
- Polylactic_acid 7y ago
  People already do this. You can submit your sitemap to bing and google. But I doubt you will get anyone to bother with a small player or p2p service.

bawolff 7y ago

I feel like there is either 2 possibilities: Small number of indexes (like HotBot before google ruled the internet). I'm not sure an oligopoly is that much better than a monopoly Or you have a lot of indexes. But then the system scales really poorly. All the indexes have to handle a large number of queries and be hugely powerful, and all the clients have to spend a ton of time querrying and aggregating them.

OJFord 7y ago

Yes! And of course a big 'enough' publisher like Medium would be incentivised to provide its own crawler implementing the standard interface, for SEO.

tobylane 7y ago

The winner will be the vertical integrator, not the best quality. People will pick the quantitative measure (speed) over anything qualitative.