5 ms·

is the size of the common Web already way too large to play catch up against google/bing at this point? my dream, is a distributed/p2p index. each browser cont

by hirako2000 5y ago

is the size of the common Web already way too large to play catch up against google/bing at this point? my dream, is a distributed/p2p index. each browser contribute to storing part of the overall index, and handle queries coming from other users so that how to fund huge data centers never become a question.

daoudc 5y ago
Check out YaCy. It's great if you're happy with slow search!
- jesprenj 5y ago
  I had problems with YaCy. Slow search and slow crawling. Regarding search I think it could be improved if instead of HTTP requests to other peers a more efficient protocol (UDP) could be used. Regarding crawling it's quite possible that doing this in Java and running it on a Rock64 may not be the best combination (: It started OOMing after some days.
- charcircuit 5y ago
  YaCy's results weren't great and there was the raking was very bad letting sides game it by just spamming a ton of keywords.
dotancohen 5y ago
> is the size of the common Web already way too large to play catch up against google/bing at this point? Probably. But I would prefer a search engine that didn't search the whole web. I would prefer a search engine that searched the sites related to fields that I'm interested in. So I would pay for or donate to a search engine that provided me good results in e.g. software development. They could add additional fields as demand warrants, so long as quality as maintained. I would even like to see a faceting feature, so I could search for e.g. Matrix and get results on the mathematical concept when need be without having to wade through movie review or fiddle with magic search keywords.
- hirako2000 5y ago
  not searching the whole Web makes sense, except that it isn't clear what belongs to your field of interest and what doesn't. should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? and what if you want to search about pottery that Sunday morning, turn to Google? I think segmenting search results by topics is a useful consumer query feature, not sure segmenting what gets indexed would provide a useful service other than covering niches hence not really fulfilling a web search engine. I find the idea less ambitious so maybe that's how an open search engine should approach the problem, federation of hosts could, cover the whole Web eventually.
  - dotancohen 5y ago
    > should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? Yes, because that author's pages wouldn't even be fetched at the point of development that we are discussing. > and what if you want to search about pottery that Sunday morning, turn to Google? Yes, Google still exists. Why not?
    - hirako2000 5y ago
      why not Google? What I find most beneficial in a search engine alternative is to be able to fetch results or topics I'm less knowledgeable about. I'm somewhat fine with very commercial search engines when searching for something within a field I'm familiar with, e.g I if search for "kubernetes configmap best practice", skimming infomercial sites, ads, junk is trivial. if I search for "baby milk safety" I have no clue how to digest the results list, I'm totally unfamiliar with the media outlets catering to parenting, nutrition or food health, I know absolutely no authors in the field, no renowned media outlets for tips, not even brand reputations to make a somewhat informed decision on what to skim and what to read with attention. But I don't disagree an engine focusing on specific fields provides great value. and perhaps that's where it should start to have a chạnce to win.

daoudc 5y ago

Check out YaCy. It's great if you're happy with slow search!

jesprenj 5y ago
I had problems with YaCy. Slow search and slow crawling. Regarding search I think it could be improved if instead of HTTP requests to other peers a more efficient protocol (UDP) could be used. Regarding crawling it's quite possible that doing this in Java and running it on a Rock64 may not be the best combination (: It started OOMing after some days.
charcircuit 5y ago
YaCy's results weren't great and there was the raking was very bad letting sides game it by just spamming a ton of keywords.

dotancohen 5y ago

> is the size of the common Web already way too large to play catch up against google/bing at this point? Probably. But I would prefer a search engine that didn't search the whole web. I would prefer a search engine that searched the sites related to fields that I'm interested in. So I would pay for or donate to a search engine that provided me good results in e.g. software development. They could add additional fields as demand warrants, so long as quality as maintained. I would even like to see a faceting feature, so I could search for e.g. Matrix and get results on the mathematical concept when need be without having to wade through movie review or fiddle with magic search keywords.

hirako2000 5y ago
not searching the whole Web makes sense, except that it isn't clear what belongs to your field of interest and what doesn't. should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? and what if you want to search about pottery that Sunday morning, turn to Google? I think segmenting search results by topics is a useful consumer query feature, not sure segmenting what gets indexed would provide a useful service other than covering niches hence not really fulfilling a web search engine. I find the idea less ambitious so maybe that's how an open search engine should approach the problem, federation of hosts could, cover the whole Web eventually.
- dotancohen 5y ago
  > should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? Yes, because that author's pages wouldn't even be fetched at the point of development that we are discussing. > and what if you want to search about pottery that Sunday morning, turn to Google? Yes, Google still exists. Why not?
  - hirako2000 5y ago
    why not Google? What I find most beneficial in a search engine alternative is to be able to fetch results or topics I'm less knowledgeable about. I'm somewhat fine with very commercial search engines when searching for something within a field I'm familiar with, e.g I if search for "kubernetes configmap best practice", skimming infomercial sites, ads, junk is trivial. if I search for "baby milk safety" I have no clue how to digest the results list, I'm totally unfamiliar with the media outlets catering to parenting, nutrition or food health, I know absolutely no authors in the field, no renowned media outlets for tips, not even brand reputations to make a somewhat informed decision on what to skim and what to read with attention. But I don't disagree an engine focusing on specific fields provides great value. and perhaps that's where it should start to have a chạnce to win.