4 ms·
Ask HN: What features would you like to see in a P2P search engine?
I have a long weekend and I am thinking of putting online a p2p search engine for the web. Searching the web is too large for me or anyone to crawl by themselves but a group of people could do text-search on the sites of interest to us.
Would you use a p2p search engine? What features would you like to see in a peer-to-peer search engine whose nodes crawl the web and run search queries?
- eimrine 4y agoability to find cp, of course.
- logicallee 4y agoI hope you will agree it is safer for everyone, including the p2p network and all users, if we apply a global list of exclusions for content like that. There is a list maintained by NCMEC and other similar lists can be added. Thanks for bringing this issue up.
- speedgoose 4y agoSome concepts about privacy and perhaps features to prevent being a tool for child pornography and other illegal and immoral content.
- logicallee 4y agoYes, a global list of exclusions for content like that such as the list maintained by NCMEC would account for this.
- _trampeltier 4y agoSome years ago, I played a but with yazy websearch (yazy.net). Yazy cannot index Tor / onion sites, so maybe that.
- r721 4y agoCreating a not-Java clone of https://en.wikipedia.org/wiki/YaCy https://en.wikipedia.org/wiki/YaCy makes sense, maybe? Another feature I think could be important is integrating https://en.wikipedia.org/wiki/Common_Crawl https://en.wikipedia.org/wiki/Common_Crawl datasets.
- logicallee 4y agoWhat language would you prefer to see a P2P search engine written in? The main contenders are: * Python, which is very easy to use, with a large ecosystem, but also has a larger installation size and would require users to install a few dependencies. * Go, which has some existing p2p libraries and compiles to a small and very fast executable, but is more difficult to implement fancy higher-level data structures or use libraries on. 2. Thank you, https://en.wikipedia.org/wiki/Common_Crawl https://en.wikipedia.org/wiki/Common_Crawl dataset is an interesting dataset. At 320 terabytes it is too large for any one user, but if a few thousand people divided up the tasks they could run interesting queries as part of a p2p network. It would be something to consider including as the network got larger.
- r721 4y ago1) Both are ok imho, I just thought there was a period in 00s when Java was popular for various apps, even for some YouTube downloader you needed it to be installed, and now I didn't see requirements like that for ages. YaCy started in 2003, and I think the language could be one of the reasons the interest in this project waned. 2) I didn't look into Common Crawl closely, but I think they maybe have smaller sample datasets for testing or home projects? It just makes sense to use freely available crawling data which already exists and not wait for people to join your project. P.S. Previous YaCy discussions on HN: https://news.ycombinator.com/from?site=yacy.net https://news.ycombinator.com/from?site=yacy.net , there are some feature requests in comments probably.