6 ms·

YaCy: Decentralized Web Search

136 points by okasaki 7y ago

tsukurimashou 7y ago
interesting, has anybody here tried to use it? I'd like to hear some user experience!
- ravenstine 7y ago
  I tried using it seriously 5 years ago. Things might have changed since then, though. All I remember is that it was tremendously slow and the results were pretty much garbage. It's an interesting project, nonetheless. EDIT: They now have an online demo that doesn't require any installation. After waiting about a minute for peers to connect, queries ran relatively "fast" each time. That's an improvement. But the results are still junk, IMO. https://yacy.searchlab.eu/Status.html https://yacy.searchlab.eu/Status.html
- Kaiyou 7y ago
  I gave it a try in 2017. I could live with it being really slow, but it didn't have any useful results for anything, so I moved on. Really like the idea, though.
  - tombert 7y ago
    Had a similar experience back in 2015 or so; I think that the idea of a decentralized web-scraper actually makes a lot of sense, but it just didn't seem to give me a result that was useful, and actually seemed to give a lot of porn results for perfectly innocuous searches. I ended up going back to DDG.
- mippenhappen 7y ago
  I played around with that years ago. Was absolutely useless at the time. Honestly doubt it has gotten much better.
slenk 7y ago
I wonder how it differs from Searx. Have been running a Searx instance for myself for a few months now and i love it.
- nindwen 7y ago
  Searx aggregates different search engines. YaCy is peer-to-peer engine that can run its own crawlers to index content, and query other peers for their content. Looks like Searx has YaCy-integration available, so you could run a local YaCy-node and get its results added to your Searx results.
  - slenk 7y ago
    Good to know. I might have to do that. Thank you.
deleted 7y ago
[deleted]
haolez 7y ago
What's the opt-in censorship story with YaCy? I don't want to accidentally find gore or dark stuff in general.
ivarv 7y ago
I think the value of this kind of project isn't necessarily to use a general public instance, but instead to train your own instance. e.g. Imagine collecting your search history into a search corpus. Once you're set up, then running your search queries across not only your node but across a trusted network of peers as well. Yes, you'd be in a filter bubble, but the results would be highly curated.
- comboy 7y ago
  That is the only way to decentralize given reality of spam and incentives to manipulate results. "trusted network of peers" is the key here. In my opinion in order for it to work trust should be weighted. And weights should be update-able based on results you want or don't want to see. Also, if we would get this decentralized web of trust with weights it's like the Internet reinvented. Thanks to small world effect you have access to reviews you can actually trust, your personal trust to ebay seller, potential business partner or any politician... When I say that is the only way, it is like a fact to me. I've spend a long time thinking about it and I'd be delighted to be proven wrong.
dana321 7y ago
I have tried this extensively in the past around a year ago, unfortunately its written in Java and suffers performance issues with GC when you start pushing it hard, the whole system crawls and the performance is erratic. I had to uninstall it as it was killing my 8-core 5960x server. The results can be ok if you push it towards the sites you enjoy reading, but apart from that..
- numpad0 7y ago
  UX wise it’s also n-gram with n>2 and only works well with Indo-European languages.
- Polylactic_acid 7y ago
  I found the results to be utterly useless. I wasn't able to perform a single search that came back with sane results. I searched "youtube" and the first few pages were random blogs. I couldn't not find "youtube.com" anywhere in the results.
gambler 7y ago
P2P is probably the best solution for search in the long run, but here is how it can be decentralized in the interim. Separate crawlers, indexers and search websites. One search website could go to several indexers. One indexer could get info from several crawlers. In today's internet this makes so much more sense than everything being run by the same company. For example, you could have a crawler that specializes at searching Medium.com. You could have another one that specializes in searching just computer game websites. The data from both are aggregated by indexer A and indexer B. Indexers would rank content using different algorithms. When user opens some search website and types a query, the website pre-processes the query and sends it to both indexers. Both return the best results they can. The website then aggregates them into a single page and shows to the user. With this system there will be competition at every level. And a startup could conceivably build a node in this infrastructure without tackling the entire thing at the same time (like you have to do right now). You think website X has bad interface? You can build your own interface and buy information from existing indexers (which you chose). You think you can do better job at ranking results? You can build your own indexer and sell your results to pre-existing websites. You think some niche area of the web is ignored? You can build your own crawler and sell your results to one or several indexers. If someone does a bad job, the next level of the system has incentives (and freedom) to drop them. This includes the consumer of search results.
- kleer001 7y ago
  Now THAT sounds like a fertile ecosystem.
- MadWombat 7y ago
  > In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on HN, so I agree with you that this would be a really neat to have an ecosystem like this. But the question is how do you make money? What is your incentive to run a crawler? What is your incentive to run an indexer? People are used to the idea that web search is free, so you search sites cannot charge people money. It also means that they cannot buy indexer services.
  - 7y ago
oever 7y ago
Distributed search is great to make searching less dependent on a few big players. But if many more bots start crawling your site, the load and waste of energy can get very high. Sitemaps help a bit, but site needs to be crawled completely initially and changed pages need to be retrieved. It'd be more efficient if there were a standard by which websites could create and publish their own index. Web search engines could collate the pages and add page ranks. The correctness of the index can be tested by randomly sampling a few pages with a bias to the highest rated pages.
stewofkc 7y ago
I'm not sure I really understand the importance of decentralized web search...yet. Why not just use something like Ghostpeek (https://ghostpeek.com https://ghostpeek.com) or DuckDuckGo (duckduckgo.com), which are ethical and trustworthy (for now)? Is the purpose of YaCy to prevent one company from having all of your search data or to provide better results?
dang 7y ago
A thread from 2016: https://news.ycombinator.com/item?id=12433010 https://news.ycombinator.com/item?id=12433010 2014: https://news.ycombinator.com/item?id=8746883 https://news.ycombinator.com/item?id=8746883 2011: https://news.ycombinator.com/item?id=3288586 https://news.ycombinator.com/item?id=3288586 (Reposts are fine after a year or so; links like this are just to supply more reading for the curious.)
bawolff 7y ago
I wish there was more technical details or a whitepaper. All i can see is some handwaving about dht. But full text search is not really naturally applicable to a DHT approach. So what exactly do they do? How does what they do scale? How is work broken up between dht nodes (what exactly is being used as the hash key in the distributed hash table)? How does it deal with malicious nodes (SEO is a big thing in search)? Is the division of resources between dht nodes "fair"? How are results ranked?
flokie 7y ago
Query "best pizza place" Result "Hardcore teen sex" Why is this trending?
- deleted 7y ago
  [deleted]
marknadal 7y ago
YaCy has been around for a long time, no? What are other projects in this space? I know Martti Malmi (Satoshi's 1st Bitcoin contributor) is doing some stuff (no tokens) in this space. Anybody else/know/have links/talks/articles on other (no blockchain) decentralized search?
DethNinja 7y ago
What if some bad actor decides to harm the network by listing illegal links? How network secures itself against that?
- Polylactic_acid 7y ago
  Other networks have lists of banned hashes. You can just hash the banned domains/links and users can subscribe to ban lists. You could even distribute them in plain text tbh. The issue is not about having the links, but accidentally clicking on them.

136 points by okasaki 7y ago

tsukurimashou 7y ago

interesting, has anybody here tried to use it? I'd like to hear some user experience!

ravenstine 7y ago
I tried using it seriously 5 years ago. Things might have changed since then, though. All I remember is that it was tremendously slow and the results were pretty much garbage. It's an interesting project, nonetheless. EDIT: They now have an online demo that doesn't require any installation. After waiting about a minute for peers to connect, queries ran relatively "fast" each time. That's an improvement. But the results are still junk, IMO. https://yacy.searchlab.eu/Status.html https://yacy.searchlab.eu/Status.html
Kaiyou 7y ago
I gave it a try in 2017. I could live with it being really slow, but it didn't have any useful results for anything, so I moved on. Really like the idea, though.
- tombert 7y ago
  Had a similar experience back in 2015 or so; I think that the idea of a decentralized web-scraper actually makes a lot of sense, but it just didn't seem to give me a result that was useful, and actually seemed to give a lot of porn results for perfectly innocuous searches. I ended up going back to DDG.
mippenhappen 7y ago
I played around with that years ago. Was absolutely useless at the time. Honestly doubt it has gotten much better.

slenk 7y ago

I wonder how it differs from Searx. Have been running a Searx instance for myself for a few months now and i love it.

nindwen 7y ago
Searx aggregates different search engines. YaCy is peer-to-peer engine that can run its own crawlers to index content, and query other peers for their content. Looks like Searx has YaCy-integration available, so you could run a local YaCy-node and get its results added to your Searx results.
- slenk 7y ago
  Good to know. I might have to do that. Thank you.

deleted 7y ago

[deleted]

haolez 7y ago

What's the opt-in censorship story with YaCy? I don't want to accidentally find gore or dark stuff in general.

ivarv 7y ago

I think the value of this kind of project isn't necessarily to use a general public instance, but instead to train your own instance. e.g. Imagine collecting your search history into a search corpus. Once you're set up, then running your search queries across not only your node but across a trusted network of peers as well. Yes, you'd be in a filter bubble, but the results would be highly curated.

comboy 7y ago
That is the only way to decentralize given reality of spam and incentives to manipulate results. "trusted network of peers" is the key here. In my opinion in order for it to work trust should be weighted. And weights should be update-able based on results you want or don't want to see. Also, if we would get this decentralized web of trust with weights it's like the Internet reinvented. Thanks to small world effect you have access to reviews you can actually trust, your personal trust to ebay seller, potential business partner or any politician... When I say that is the only way, it is like a fact to me. I've spend a long time thinking about it and I'd be delighted to be proven wrong.

dana321 7y ago

I have tried this extensively in the past around a year ago, unfortunately its written in Java and suffers performance issues with GC when you start pushing it hard, the whole system crawls and the performance is erratic. I had to uninstall it as it was killing my 8-core 5960x server. The results can be ok if you push it towards the sites you enjoy reading, but apart from that..

numpad0 7y ago
UX wise it’s also n-gram with n>2 and only works well with Indo-European languages.
Polylactic_acid 7y ago
I found the results to be utterly useless. I wasn't able to perform a single search that came back with sane results. I searched "youtube" and the first few pages were random blogs. I couldn't not find "youtube.com" anywhere in the results.

gambler 7y ago

P2P is probably the best solution for search in the long run, but here is how it can be decentralized in the interim. Separate crawlers, indexers and search websites. One search website could go to several indexers. One indexer could get info from several crawlers. In today's internet this makes so much more sense than everything being run by the same company. For example, you could have a crawler that specializes at searching Medium.com. You could have another one that specializes in searching just computer game websites. The data from both are aggregated by indexer A and indexer B. Indexers would rank content using different algorithms. When user opens some search website and types a query, the website pre-processes the query and sends it to both indexers. Both return the best results they can. The website then aggregates them into a single page and shows to the user. With this system there will be competition at every level. And a startup could conceivably build a node in this infrastructure without tackling the entire thing at the same time (like you have to do right now). You think website X has bad interface? You can build your own interface and buy information from existing indexers (which you chose). You think you can do better job at ranking results? You can build your own indexer and sell your results to pre-existing websites. You think some niche area of the web is ignored? You can build your own crawler and sell your results to one or several indexers. If someone does a bad job, the next level of the system has incentives (and freedom) to drop them. This includes the consumer of search results.

kleer001 7y ago
Now THAT sounds like a fertile ecosystem.
MadWombat 7y ago
> In today's internet this makes so much more sense than everything being run by the same company I have expressed similar ideas in other discussions here on HN, so I agree with you that this would be a really neat to have an ecosystem like this. But the question is how do you make money? What is your incentive to run a crawler? What is your incentive to run an indexer? People are used to the idea that web search is free, so you search sites cannot charge people money. It also means that they cannot buy indexer services.
- 7y ago

oever 7y ago

Distributed search is great to make searching less dependent on a few big players. But if many more bots start crawling your site, the load and waste of energy can get very high. Sitemaps help a bit, but site needs to be crawled completely initially and changed pages need to be retrieved. It'd be more efficient if there were a standard by which websites could create and publish their own index. Web search engines could collate the pages and add page ranks. The correctness of the index can be tested by randomly sampling a few pages with a bias to the highest rated pages.

stewofkc 7y ago

I'm not sure I really understand the importance of decentralized web search...yet. Why not just use something like Ghostpeek (https://ghostpeek.com https://ghostpeek.com) or DuckDuckGo (duckduckgo.com), which are ethical and trustworthy (for now)? Is the purpose of YaCy to prevent one company from having all of your search data or to provide better results?

dang 7y ago

A thread from 2016: https://news.ycombinator.com/item?id=12433010 https://news.ycombinator.com/item?id=12433010 2014: https://news.ycombinator.com/item?id=8746883 https://news.ycombinator.com/item?id=8746883 2011: https://news.ycombinator.com/item?id=3288586 https://news.ycombinator.com/item?id=3288586 (Reposts are fine after a year or so; links like this are just to supply more reading for the curious.)

bawolff 7y ago

I wish there was more technical details or a whitepaper. All i can see is some handwaving about dht. But full text search is not really naturally applicable to a DHT approach. So what exactly do they do? How does what they do scale? How is work broken up between dht nodes (what exactly is being used as the hash key in the distributed hash table)? How does it deal with malicious nodes (SEO is a big thing in search)? Is the division of resources between dht nodes "fair"? How are results ranked?

flokie 7y ago

Query "best pizza place" Result "Hardcore teen sex" Why is this trending?

deleted 7y ago
[deleted]

marknadal 7y ago

YaCy has been around for a long time, no? What are other projects in this space? I know Martti Malmi (Satoshi's 1st Bitcoin contributor) is doing some stuff (no tokens) in this space. Anybody else/know/have links/talks/articles on other (no blockchain) decentralized search?

DethNinja 7y ago

What if some bad actor decides to harm the network by listing illegal links? How network secures itself against that?

Polylactic_acid 7y ago
Other networks have lists of banned hashes. You can just hash the banned domains/links and users can subscribe to ban lists. You could even distribute them in plain text tbh. The issue is not about having the links, but accidentally clicking on them.