5 ms·

The only way to have privacy in search is not to give your queries to a 3rd party i.e. you need decentralised search. The basic principle of decentralised sear

by harryf 9y ago

The only way to have privacy in search is not to give your queries to a 3rd party i.e. you need decentralised search. The basic principle of decentralised search is an individual only needs a tiny fraction of all knowledge available, so the solution needs to focus on indexing their interests and social connections - see http://www.mathaware.org/mam/04/essays/smallworld.html http://www.mathaware.org/mam/04/essays/smallworld.html It has been tried already. There's YaCy https://yacy.net/en/index.html https://yacy.net/en/index.html but the UX is horrible ( https://yacy.net/en/Screenshots.html https://yacy.net/en/Screenshots.html ) - not for the mainstream software consumer. Then there's http://www.faroo.com/ http://www.faroo.com/ ( see https://en.wikipedia.org/wiki/FAROO https://en.wikipedia.org/wiki/FAROO ) which has been ticking along since 2005 and tried to appeal to consumers but arguably aliened early adopters most by targeting Windows. They seem to have stalled otherwise ( http://www.faroo.com/hp/p2p/history.html http://www.faroo.com/hp/p2p/history.html) but it still works as a service I believe. Lastly there's https://www.bitclave.com/ https://www.bitclave.com/ - the new kid on the block. They seem to be aligning themselves with crypto-currencies somehow - perhaps there's a path there to monetisation - see https://www.cryptocoinsnews.com/bitclave-search-engine-taps-blockchain-decentralize-advertising-ico-planned/ https://www.cryptocoinsnews.com/bitclave-search-engine-taps-... and https://hackernoon.com/bitclave-the-decentralised-search-engine-257511813599 https://hackernoon.com/bitclave-the-decentralised-search-eng... - one to watch.

rahiel 9y ago
> The only way to have privacy in search is not to give your queries to a 3rd party i.e. you need decentralised search. Another way is by connecting to a search engine within the Tor network, like DuckDuckGo's hidden service: https://3g2upl4pq6kufc4m.onion https://3g2upl4pq6kufc4m.onion
jasode 9y ago
>The basic principle of decentralised search is an individual only needs a tiny fraction of all knowledge available, That may be a reasonable assumption but it omits a major tradeoff: some types of "knowledge" require the search engine to "see" the entire index and synthesizing such knowledge cannot be delegated to decentralized nodes. To reduce the search engine tradeoff to a simple math analogy: You can "decentralize" the calculation of a SUM to individual nodes. However, you can't decentralize the calculation of an AVERAGE. Divide-and-conquer strategy of calculating an average will give the wrong result.[1] Some searches are like "SUMS", and some are like "AVERAGES". If a search is pure text retrieval without regards to relative weighting or ranking, decentralized nodes with each holding partial segment of the index will work. (E.g. you want to retrieve a chili recipe so your search query hits the decentralized node that has recipes starting with "C" or whatever hash algorithm.) That's a "SUM" type of problem. However, if you want to find the top 10 ranked recipes by links, the decentralized nodes can't answer that. That's a "calculate the AVERAGE" type of problem. Also, determining which webpages in the index are "fake news" or SEO "link farms" and other "bad actors" is also an "AVERAGE" type of problem. The machine learning algorithms need to see the whole index (or pedantically, a very large portion of it) to "learn" what "fake news" looks like. Google's ability to comb their entire index in a centralized place lets them discover new insights to evolve the Pagerank algorithm from Panda->Penguin->Hummingbird->etc.[2] Yes, you gain more privacy with decentralized or p2p search, but you also lose the ability to search on certain types of queries. [1] https://math.stackexchange.com/questions/115091/is-the-average-of-the-averages-equal-to-the-average-of-all-the-numbers-originall https://math.stackexchange.com/questions/115091/is-the-avera... [2] https://conversion-hub.com/wp-content/uploads/2016/08/Comparison.bmp https://conversion-hub.com/wp-content/uploads/2016/08/Compar...

by harryf 9y ago

rahiel 9y ago

> The only way to have privacy in search is not to give your queries to a 3rd party i.e. you need decentralised search. Another way is by connecting to a search engine within the Tor network, like DuckDuckGo's hidden service: https://3g2upl4pq6kufc4m.onion https://3g2upl4pq6kufc4m.onion

jasode 9y ago

>The basic principle of decentralised search is an individual only needs a tiny fraction of all knowledge available, That may be a reasonable assumption but it omits a major tradeoff: some types of "knowledge" require the search engine to "see" the entire index and synthesizing such knowledge cannot be delegated to decentralized nodes. To reduce the search engine tradeoff to a simple math analogy: You can "decentralize" the calculation of a SUM to individual nodes. However, you can't decentralize the calculation of an AVERAGE. Divide-and-conquer strategy of calculating an average will give the wrong result.[1] Some searches are like "SUMS", and some are like "AVERAGES". If a search is pure text retrieval without regards to relative weighting or ranking, decentralized nodes with each holding partial segment of the index will work. (E.g. you want to retrieve a chili recipe so your search query hits the decentralized node that has recipes starting with "C" or whatever hash algorithm.) That's a "SUM" type of problem. However, if you want to find the top 10 ranked recipes by links, the decentralized nodes can't answer that. That's a "calculate the AVERAGE" type of problem. Also, determining which webpages in the index are "fake news" or SEO "link farms" and other "bad actors" is also an "AVERAGE" type of problem. The machine learning algorithms need to see the whole index (or pedantically, a very large portion of it) to "learn" what "fake news" looks like. Google's ability to comb their entire index in a centralized place lets them discover new insights to evolve the Pagerank algorithm from Panda->Penguin->Hummingbird->etc.[2] Yes, you gain more privacy with decentralized or p2p search, but you also lose the ability to search on certain types of queries. [1] https://math.stackexchange.com/questions/115091/is-the-average-of-the-averages-equal-to-the-average-of-all-the-numbers-originall https://math.stackexchange.com/questions/115091/is-the-avera... [2] https://conversion-hub.com/wp-content/uploads/2016/08/Comparison.bmp https://conversion-hub.com/wp-content/uploads/2016/08/Compar...