5 ms·

YaCy: a free distributed search engine

101 points by e19293001 10y ago

jacquesm 10y ago
This has been posted to HN many times before, this was the most active thread: https://news.ycombinator.com/item?id=8746883 https://news.ycombinator.com/item?id=8746883
- r721 10y ago
  It looks like there was a big update recently: http://yacy.net/release_notes/YaCy_Release_1_90.html http://yacy.net/release_notes/YaCy_Release_1_90.html (Jul 2016)
singularity2001 10y ago
"Your peer cannot accept remote crawls because you need senior or principal peer status for that!" So the average user cannot contribute to building the crawl index?
- executesorder66 10y ago
  That is just the default setting, to prevent using massive bandwidth. You can change the setting to allow for that.
  - singularity2001 10y ago
    couldn't find it
    - executesorder66 10y ago
      Sorry, I hadn't used it in a while. You need to open up the port. See here: http://www.yacy-websearch.net/wiki/index.php/En:FAQ#My_peer_says_it_runs_in_.27Junior_Mode.27._How_can_I_run_it_in_Senior_Mode.3F http://www.yacy-websearch.net/wiki/index.php/En:FAQ#My_peer_...
singularity2001 10y ago
search "learning to learn gradient descent by gradient descent" The following words are stop-words and had been excluded from the search: [by, to]. Results: 1) Linux-Kernel Archive: By Thread 2) Series Cinematography by 3) Key to Citations As they are bold, "by" and "to" seem to get boosted!!
bobajeff 10y ago
Peer-to-peer Information Retrieval is one of those old research topics that face the difficult to solve problem of scalability. YaCy is yet an early contributor to this research but we've yet to make something to my knowledge that gives comparable results to centralised search engines, is fast and scales well.
- fiatjaf 10y ago
  Any search engine is good enough, but no one, centralized or decentralized, is even close to Google. Google is our wonder and our curse.
  - p4bl0 10y ago
    That's why I use startpage.com, it returns Google results while protecting your privacy :).
    - DarkLinkXXXX 10y ago
      I use searx.me for that. It's open-source, and it's a meta-search engine that returns results from more than just Google.
  - bobajeff 10y ago
    >Any search engine is good enough That's simply not true. Things we take for granted like true multi-query search and sub-second results speeds simply aren't seen in distributed search engines.
  - eggie 10y ago
    I actually think that the duck (ddg.gg) is better. I now find it difficult to use google, which feels clunky and capricious. I make thousands of queries a day for work, so I really can't excuse poor performance. For a while I would switch back to google when my searches failed and I wanted to check if the duck was getting me equivalent or better answers. In the past year and a half I've completely stopped. It is simply a high-quality product that exceeds google in what it offers. Maybe you'll respond that "it just doesn't 'feel' high quality". OK, I can't argue with that because it's your opinion... but as long as we collectively decide to believe this fantasy we will continue to be locked into the search juggernaut. You can help improve the diversity of options by using something different and get to use a better product. The only thing that I can't do on the duck is search academic papers. Google scholar is still the best thing of its type.
steaminghacker 10y ago
Wow! I had the same idea over 10 years ago. I'm glad it's being done. My idea was using a giant P2P indexbase. However, I read the tech page, but it doesn't explain enough. For example, for this to work, there would have to be some kind of "information routing" whereby queries within certain information domains would be routed to machines. I can't see how this is being done. Because you can't have the whole index on each machine, therefore, how does it know where to route a given query? Additionally, peer nodes would have to crawl to a certain criteria; either subject based, or perhaps geographically. Otherwise you'd just end up with the same stuff everywhere. If there's some more explanation, please link. thanks.
CaptSpify 10y ago
So, has anyone used it? If so, how is it? The last time I looked at this (long, long ago), it was pretty much poisoned by penis-enlargement and drug ads. I'm hoping it's gotten better since then.
ShirsenduK 10y ago
This page took 25GB of RAM on Safari!

101 points by e19293001 10y ago

This has been posted to HN many times before, this was the most active thread: https://news.ycombinator.com/item?id=8746883 https://news.ycombinator.com/item?id=8746883

r721 10y ago
It looks like there was a big update recently: http://yacy.net/release_notes/YaCy_Release_1_90.html http://yacy.net/release_notes/YaCy_Release_1_90.html (Jul 2016)

singularity2001 10y ago

"Your peer cannot accept remote crawls because you need senior or principal peer status for that!" So the average user cannot contribute to building the crawl index?

executesorder66 10y ago
That is just the default setting, to prevent using massive bandwidth. You can change the setting to allow for that.
- singularity2001 10y ago
  couldn't find it
  - executesorder66 10y ago
    Sorry, I hadn't used it in a while. You need to open up the port. See here: http://www.yacy-websearch.net/wiki/index.php/En:FAQ#My_peer_says_it_runs_in_.27Junior_Mode.27._How_can_I_run_it_in_Senior_Mode.3F http://www.yacy-websearch.net/wiki/index.php/En:FAQ#My_peer_...

singularity2001 10y ago

search "learning to learn gradient descent by gradient descent" The following words are stop-words and had been excluded from the search: [by, to]. Results: 1) Linux-Kernel Archive: By Thread 2) Series Cinematography by 3) Key to Citations As they are bold, "by" and "to" seem to get boosted!!

bobajeff 10y ago

Peer-to-peer Information Retrieval is one of those old research topics that face the difficult to solve problem of scalability. YaCy is yet an early contributor to this research but we've yet to make something to my knowledge that gives comparable results to centralised search engines, is fast and scales well.

fiatjaf 10y ago
Any search engine is good enough, but no one, centralized or decentralized, is even close to Google. Google is our wonder and our curse.
- p4bl0 10y ago
  That's why I use startpage.com, it returns Google results while protecting your privacy :).
  - DarkLinkXXXX 10y ago
    I use searx.me for that. It's open-source, and it's a meta-search engine that returns results from more than just Google.
- bobajeff 10y ago
  >Any search engine is good enough That's simply not true. Things we take for granted like true multi-query search and sub-second results speeds simply aren't seen in distributed search engines.
- eggie 10y ago
  I actually think that the duck (ddg.gg) is better. I now find it difficult to use google, which feels clunky and capricious. I make thousands of queries a day for work, so I really can't excuse poor performance. For a while I would switch back to google when my searches failed and I wanted to check if the duck was getting me equivalent or better answers. In the past year and a half I've completely stopped. It is simply a high-quality product that exceeds google in what it offers. Maybe you'll respond that "it just doesn't 'feel' high quality". OK, I can't argue with that because it's your opinion... but as long as we collectively decide to believe this fantasy we will continue to be locked into the search juggernaut. You can help improve the diversity of options by using something different and get to use a better product. The only thing that I can't do on the duck is search academic papers. Google scholar is still the best thing of its type.

steaminghacker 10y ago

Wow! I had the same idea over 10 years ago. I'm glad it's being done. My idea was using a giant P2P indexbase. However, I read the tech page, but it doesn't explain enough. For example, for this to work, there would have to be some kind of "information routing" whereby queries within certain information domains would be routed to machines. I can't see how this is being done. Because you can't have the whole index on each machine, therefore, how does it know where to route a given query? Additionally, peer nodes would have to crawl to a certain criteria; either subject based, or perhaps geographically. Otherwise you'd just end up with the same stuff everywhere. If there's some more explanation, please link. thanks.

CaptSpify 10y ago

So, has anyone used it? If so, how is it? The last time I looked at this (long, long ago), it was pretty much poisoned by penis-enlargement and drug ads. I'm hoping it's gotten better since then.

ShirsenduK 10y ago

This page took 25GB of RAM on Safari!