3 ms·

What we definitely need is an independent, not for profit, decentralized (ideally p2p) search engine. Because of conflicts of interest, web search can not be le

What we definitely need is an independent, not for profit, decentralized (ideally p2p) search engine. Because of conflicts of interest, web search can not be left for companies.

marginalia_nu 5y ago
Decentralized search is pretty tricky to get working well. Crawling and indexing is trivial to parallelize, but to get a performant search engine you need a good ranking algorithm, and that pretty much requires a central node with access to the entire link graph.
- marcodiego 5y ago
  Sometime ago, I thought about some form of simple ranking that could give good results. I thought how about giving points for sites that are linked by a small set of "sources" like hacker news, wikipedia, stack overflow, reddit, news sites... I always wondered how such results could be. Never had the knowledge to test it.
  - marginalia_nu 5y ago
    Like so https://dkb.io/post/market-rank https://dkb.io/post/market-rank ?
  - soco 5y ago
    Those points are then given site-wide no matter what a certain page content would rank? For sintance on dev.to you have a sizeable amount of garbage/spam articles which would benefit of this...
- soco 5y ago
  Now only if the ranking could happen on my machine too, based on my (search) history... not in this life probably.
  - marginalia_nu 5y ago
    That's probably doable, but the problem is this information needs to be available upstream. Say you search for "seafood" and there are 50,000,000 results from 100,000 domains because everyone just loves making websites about clams and sushi; it's not really feasible to send you all of those results, your query would take 15 minutes and that's not a nice experience. The tricky part in search is finding the needles in the haystack. So the crux is each of the index shards needs to know the ranking to be able to send you just the most relevant results.
deleted 5y ago
[deleted]

What we definitely need is an independent, not for profit, decentralized (ideally p2p) search engine. Because of conflicts of interest, web search can not be left for companies.

marginalia_nu 5y ago

Decentralized search is pretty tricky to get working well. Crawling and indexing is trivial to parallelize, but to get a performant search engine you need a good ranking algorithm, and that pretty much requires a central node with access to the entire link graph.

marcodiego 5y ago
Sometime ago, I thought about some form of simple ranking that could give good results. I thought how about giving points for sites that are linked by a small set of "sources" like hacker news, wikipedia, stack overflow, reddit, news sites... I always wondered how such results could be. Never had the knowledge to test it.
- marginalia_nu 5y ago
  Like so https://dkb.io/post/market-rank https://dkb.io/post/market-rank ?
- soco 5y ago
  Those points are then given site-wide no matter what a certain page content would rank? For sintance on dev.to you have a sizeable amount of garbage/spam articles which would benefit of this...
soco 5y ago
Now only if the ranking could happen on my machine too, based on my (search) history... not in this life probably.
- marginalia_nu 5y ago
  That's probably doable, but the problem is this information needs to be available upstream. Say you search for "seafood" and there are 50,000,000 results from 100,000 domains because everyone just loves making websites about clams and sushi; it's not really feasible to send you all of those results, your query would take 15 minutes and that's not a nice experience. The tricky part in search is finding the needles in the haystack. So the crux is each of the index shards needs to know the ranking to be able to send you just the most relevant results.

deleted 5y ago

[deleted]