5 ms·

I have been playing with an idea to decentralize search (or, more generally, information processing). I would consist of - a publicly available hierarchical t

I have been playing with an idea to decentralize search (or, more generally, information processing). I would consist of - a publicly available hierarchical type system (similar to how classes look like in an OOP language with multiple inheritance). For example, URL is a type, or set of search terms is a type. - any number of nodes on the net. Each of them would provide a mapping from a set of input data to a set of output data (data is defined in the terms of the types). For example, Google would provide a mapping from set of search terms to URLs, but that's a very generic node. - the nodes also advertise the cost of one mapping operation. The cost is given in some kind of currency (?). - there should be routers that can find a route between a set of input types to a set of output types. The route is composed of nodes, obviously, that pass information between them and finally the output is the stuff the user wanted. - the user pays the sum of the cost needed for his query (transformation) - the user could pay nothing if some nodes have negative cost. There could be a node that takes a set of URLs (search results) and produces a set of URLs that are basically the original URLs + some advertisements. Therefore if somebody specifies that he wants a transformation from types X to types Z (and provides data of types X) under cost C, then the routers should find a set of nodes under these constraints - nodes could provide mappings that are the same as other nodes, and the routers would slowly insert them into mapping chains, and rate them based on some user feedback (this is essentially a genetic algorithm) This system could very much help good competition in the distributed information processing industry. Nowadays nobody can really compete with Google or Bing because you need to build _everything_ in one step. A distributed infrastructure like this that's based on the evolution of components would lower the barriers to entry dramatically. Also, data would be shared across participants, not owned only by the one single market leader. Ok, this is just a very naive and early idea, and it's also too big to implement in my spare time at the moment, but I think it'd be an interesting way to apply the good old Unix principles for building internet services.

kordless 12y ago
If you need a crawler for this, let me know. :)
fchollet 12y ago
How would you guarantee the trustworthiness of a node? How do you guarantee that a node is actually doing what it is supposed to and not inserting malware into its results? Reputation systems can be gamed (and will be if there's money at stake). I like the notion of trustworthy, general-purpose, publicly distributed computing: http://unenumerated.blogspot.com/2014/12/the-dawn-of-trustworthy-computing.html http://unenumerated.blogspot.com/2014/12/the-dawn-of-trustwo...
- sz4kerto 12y ago
  I don't know, maybe you cannot guarantee it -- similarly, you cannot guarantee the trustworthyness of Google, or Verisign, etc. They have good reputation in general (let aside the recent NSA stuff), therefore you trust them when you use them. Current systems can be broken as well (starting from DNS poisoning to hacking the actual service, etc.). The idea is not to fix all the problems once -- I'm mostly concerned with a) distributing responsibility, data and therefore power across more parties b) lowering barriers to entry into complex information processing, and as a result, increase innovation. A bit like a cathedral vs. bazaar approach. Again, I don't consider myself a computer wizard who could implement every aspect of this stuff; I am mostly interested in the algo that selects/evolves processing routes based on queries. But then there's the security, payment, service discovery, etc.
- dvanduzer 12y ago
  > How would you guarantee the trustworthiness of a node? With math. First you need a common consensus of what the publicly searchable corpus is at a given point in time. Computing a hash on the entire set may be impractical, so you partition the crawlspace. Dynamic content makes it tricky to validate distributed crawling, which motivates things like http://named-data.net http://named-data.net A blockchain might be useful as the public ledger for retrospective diffs when asking "What is all the currently public available data right now?"
bachback 12y ago
Very interesting. How about creating a github project for this? Would like to follow. Blockchain based DNS systems already exist. Web search mostly builds on links and DNS. If one thinks about it TLD names are pretty much fixed, where as in Namecoin they are subject to consensus. What one can do, and that's one example out of many, is create unique hashes for any content to create indestructible information.
- sz4kerto 12y ago
  Maybe in the near future I'll create some kind of skeleton, but I have to write this down properly, not just in a HN comment. Blockchain is not something that I'd use for these kind of stuff as it trades computational power for trust. As it came up somewhere else: the whole Bitcoin accounting could be run on a Raspberry Pi if we trusted the guy who owns the RPi -- but people who want to use BC don't trust anybody that much. Current DNS systems are based on trust and they work reasonably well, and I think while the blockchain paradigm is fashionable and fascinating, it does not need to be applied to every single kind of problem in distributed computing.
  - bachback 12y ago
    With Namecoin anyone can design new naming protocols based on consensus. That's not possible with ICANN's DNS (the Internet is centralized in this regard - nobody can change the semantics of ".com"). New naming protocols are a huge deal => for searching & finding any information, for Identification, and more. Namecoin did not get it right, and PoW style blockchains are too complicated. But potentially one gets things like single-sign on, integrated micropayments, and things which are hard to imagine (smart contracts, etc.).
miguelrochefort 12y ago
It already exists and it is called Ethereum. https://www.ethereum.org/ https://www.ethereum.org/
- drdeca 12y ago
  While I assume one could, I'm not sure it would be easy to implement web search in Ethereum? First of all, you would need a way for a contract to be able to access webpages to get information about links. I the contract could pay people to give it the content of a webpage? I'm not sure how it would verify it for non https websites, without getting copies from multiple users, which would cause problems if the webpage was not static. Second of all, the memory to store what pages link to what pages would, I think, be expensive for contract storage. Maybe if it used swarm? I guess it would use swarm. Still, the contract would have to have a short running time (in order to be relatively inexpensive). Maybe if instead of the contract producing the results, it would have others create the results, and then verify them in some way, and reward those that created the results? I don't know. I don't really know my stuff, but it seems to me that implementing a decentralized search engine with Ethereum would be fairly difficult.
  - deleted 12y ago
    [deleted]
kbody 12y ago
Check http://yacy.net/ http://yacy.net/
bobajeff 12y ago
I believe the field of research your referring to is called Peer-to-Peer Information Retrieval. Doing a search on it will bring up many papers published on it going through the various attempts, issues with and theories on building such a system that's sufficiently good enough to be competitive with a centralized search engine.

kordless 12y ago

If you need a crawler for this, let me know. :)

fchollet 12y ago

How would you guarantee the trustworthiness of a node? How do you guarantee that a node is actually doing what it is supposed to and not inserting malware into its results? Reputation systems can be gamed (and will be if there's money at stake). I like the notion of trustworthy, general-purpose, publicly distributed computing: http://unenumerated.blogspot.com/2014/12/the-dawn-of-trustworthy-computing.html http://unenumerated.blogspot.com/2014/12/the-dawn-of-trustwo...

sz4kerto 12y ago
I don't know, maybe you cannot guarantee it -- similarly, you cannot guarantee the trustworthyness of Google, or Verisign, etc. They have good reputation in general (let aside the recent NSA stuff), therefore you trust them when you use them. Current systems can be broken as well (starting from DNS poisoning to hacking the actual service, etc.). The idea is not to fix all the problems once -- I'm mostly concerned with a) distributing responsibility, data and therefore power across more parties b) lowering barriers to entry into complex information processing, and as a result, increase innovation. A bit like a cathedral vs. bazaar approach. Again, I don't consider myself a computer wizard who could implement every aspect of this stuff; I am mostly interested in the algo that selects/evolves processing routes based on queries. But then there's the security, payment, service discovery, etc.
dvanduzer 12y ago
> How would you guarantee the trustworthiness of a node? With math. First you need a common consensus of what the publicly searchable corpus is at a given point in time. Computing a hash on the entire set may be impractical, so you partition the crawlspace. Dynamic content makes it tricky to validate distributed crawling, which motivates things like http://named-data.net http://named-data.net A blockchain might be useful as the public ledger for retrospective diffs when asking "What is all the currently public available data right now?"

bachback 12y ago

Very interesting. How about creating a github project for this? Would like to follow. Blockchain based DNS systems already exist. Web search mostly builds on links and DNS. If one thinks about it TLD names are pretty much fixed, where as in Namecoin they are subject to consensus. What one can do, and that's one example out of many, is create unique hashes for any content to create indestructible information.

sz4kerto 12y ago
Maybe in the near future I'll create some kind of skeleton, but I have to write this down properly, not just in a HN comment. Blockchain is not something that I'd use for these kind of stuff as it trades computational power for trust. As it came up somewhere else: the whole Bitcoin accounting could be run on a Raspberry Pi if we trusted the guy who owns the RPi -- but people who want to use BC don't trust anybody that much. Current DNS systems are based on trust and they work reasonably well, and I think while the blockchain paradigm is fashionable and fascinating, it does not need to be applied to every single kind of problem in distributed computing.
- bachback 12y ago
  With Namecoin anyone can design new naming protocols based on consensus. That's not possible with ICANN's DNS (the Internet is centralized in this regard - nobody can change the semantics of ".com"). New naming protocols are a huge deal => for searching & finding any information, for Identification, and more. Namecoin did not get it right, and PoW style blockchains are too complicated. But potentially one gets things like single-sign on, integrated micropayments, and things which are hard to imagine (smart contracts, etc.).

miguelrochefort 12y ago

It already exists and it is called Ethereum. https://www.ethereum.org/ https://www.ethereum.org/

drdeca 12y ago
While I assume one could, I'm not sure it would be easy to implement web search in Ethereum? First of all, you would need a way for a contract to be able to access webpages to get information about links. I the contract could pay people to give it the content of a webpage? I'm not sure how it would verify it for non https websites, without getting copies from multiple users, which would cause problems if the webpage was not static. Second of all, the memory to store what pages link to what pages would, I think, be expensive for contract storage. Maybe if it used swarm? I guess it would use swarm. Still, the contract would have to have a short running time (in order to be relatively inexpensive). Maybe if instead of the contract producing the results, it would have others create the results, and then verify them in some way, and reward those that created the results? I don't know. I don't really know my stuff, but it seems to me that implementing a decentralized search engine with Ethereum would be fairly difficult.
- deleted 12y ago
  [deleted]

kbody 12y ago

Check http://yacy.net/ http://yacy.net/

bobajeff 12y ago

I believe the field of research your referring to is called Peer-to-Peer Information Retrieval. Doing a search on it will bring up many papers published on it going through the various attempts, issues with and theories on building such a system that's sufficiently good enough to be competitive with a centralized search engine.