4 ms·

> Google’s capture of the Web is a fait accompli. Only legislation will keep the World Wide Web from finally becoming Web by Google (TM). I really hope it's no

> Google’s capture of the Web is a fait accompli. Only legislation will keep the World Wide Web from finally becoming Web by Google (TM). I really hope it's not the case, because then Google will cancel it. /s Search is probably something where there could be creative use of the p2p decentralized technologies. If when performing a search I could ask my contacts to look in their history if they have something matching, and then they ask their contact to do the same, who ask their contact to do the same, I think we could have fast and massively parallelized trust based searches. Then you can rank the results based on how many times they appeared.

staticelf 6y ago
Google, the original author behind cancel culture /s
Spearchucker 6y ago
Somehow asking a contact for a recommendation or search result has never felt right. I have many friends I value very very much and yet I don't share their political views, I don't like their technology preferences and I sure as damn don't agree with their musicsl tastes. Right now I'll settle for a centralized search engine/service that works like Google did before all the crazy, like before Amp, and before penalizing non-mobile-optimized sites, and so on. Federated search scares me because it might constrain search results by being tainted with a localised world view.
tomschwiha 6y ago
Wouldn't this cause filter bubbles? And could open up privacy abuse because all your history is available?
alfonsodev 6y ago
>Search is probably something where there could be creative use of the p2p decentralized technologies. Has this been attempted before? does it have a name?
- teddyh 6y ago
  YaCy calls it “P2P Mode”: https://yacy.net/ https://yacy.net/
- ptx 6y ago
  Gnutella, 20 years ago: https://en.m.wikipedia.org/wiki/Gnutella#Design https://en.m.wikipedia.org/wiki/Gnutella#Design
texasbigdata 6y ago
Re: fair acompli Respectfully, not really. Arguably search is much worse now than in 2010. What if someone could come up with a simple search that could easily understand “hey this is a paywall” or “hey this is a paywall but somehow securely and with privacy I checked and you have a login to this site in your password manager” or it could do time based searches better than “last day / last week / etc”. Consumers have insatiable needs and are never satisfied. It’s not clear someone can’t create something unimagined yet that’s vastly technically superior based on solely focusing on customer experience. Even if you just assume NewEngine(tm) is venture backed and can lose money in the short run until it monetizes later it should still theoretically enable it to be closer to consumer desires than Google which has to post quarterly ad revenue figures to wall street and the SEC.
- alangibson 6y ago
  Fair enough, but I didn't base my argument on just search. They hold a commanding position in just about every significant area of the Web. A better product is not enough to dislodge them. And lets not forget that a very large portion of the population would need to hear about that product on an AMP page in their Chrome browser on their Android phone, probably following a Google search or noticing a story on Google news.
- NateEag 6y ago
  The new competitor would have to be so much better than Google that any given user has a >10% chance of switching, for the competitor to capture a remotely relevant marketshare. That seems fairly unlikely to me. Google gives you what you're looking for immediately for the average query, and anything beyond that is a power user detail not relevant to most people.
andai 6y ago
I've been thinking about this for a while. It's infeasible for any one person to run a Google-scale web crawler -- but if millions of people join in? I've used file sharing software which has the same "tree style" query forwarding. I think latency would be the big issue though, we're used to getting a result instantly -- and the right result too (Google does a lot of extra work knowing what you actually mean, not just keyword search).
- mark_l_watson 6y ago
  Common Crawl is a good resource if having crawl data potentially being from a month ago is OK for your application. Donate and use their crawl data.
alangibson 6y ago
I'm working in an architecture for a semi-decentralized, some-of-the-web crawler and search engine to try to address some of the problems with Google. Basically affinity groups maintain lists of links (sort of like social bookmarking) that are the crawl root urls. Crawling comes from Common Crawl plus updates for recency. Groups can pool resources to run their own indexes and receive a share of the profits from text display ads.
- lhnz 6y ago
  Is there anything that I can follow? I'm very interested in this -- I was considering making something similar myself.
  - alangibson 6y ago
    I'll give you a shout when I publish the first draft. From your profile you seem like a pretty easy guy to reach. :)
avasthe 6y ago
Distributed schemes don't take off. Especially this one can lead to severe privacy / security loopholes. And what about people who rarely come online / have limited bandwidth? A more realistic dream would be finding good algorithms for "webscale" search engines, that all competitors (Bing, DDG) can use such that any one of them can't have excessive advantage over the others. Secondly, human curated directories may still have future. If I search Google for interesting links or just programming blogs on specific topic, I find lot of SEO optimized low quality content instead of deep technical content so that I often have to search through HN, reddit etc.. and sort by popularity - but searching for particular topic is difficult.

staticelf 6y ago

Google, the original author behind cancel culture /s

Spearchucker 6y ago

Somehow asking a contact for a recommendation or search result has never felt right. I have many friends I value very very much and yet I don't share their political views, I don't like their technology preferences and I sure as damn don't agree with their musicsl tastes. Right now I'll settle for a centralized search engine/service that works like Google did before all the crazy, like before Amp, and before penalizing non-mobile-optimized sites, and so on. Federated search scares me because it might constrain search results by being tainted with a localised world view.

tomschwiha 6y ago

Wouldn't this cause filter bubbles? And could open up privacy abuse because all your history is available?

alfonsodev 6y ago

>Search is probably something where there could be creative use of the p2p decentralized technologies. Has this been attempted before? does it have a name?

teddyh 6y ago
YaCy calls it “P2P Mode”: https://yacy.net/ https://yacy.net/
ptx 6y ago
Gnutella, 20 years ago: https://en.m.wikipedia.org/wiki/Gnutella#Design https://en.m.wikipedia.org/wiki/Gnutella#Design

texasbigdata 6y ago

Re: fair acompli Respectfully, not really. Arguably search is much worse now than in 2010. What if someone could come up with a simple search that could easily understand “hey this is a paywall” or “hey this is a paywall but somehow securely and with privacy I checked and you have a login to this site in your password manager” or it could do time based searches better than “last day / last week / etc”. Consumers have insatiable needs and are never satisfied. It’s not clear someone can’t create something unimagined yet that’s vastly technically superior based on solely focusing on customer experience. Even if you just assume NewEngine(tm) is venture backed and can lose money in the short run until it monetizes later it should still theoretically enable it to be closer to consumer desires than Google which has to post quarterly ad revenue figures to wall street and the SEC.

alangibson 6y ago
Fair enough, but I didn't base my argument on just search. They hold a commanding position in just about every significant area of the Web. A better product is not enough to dislodge them. And lets not forget that a very large portion of the population would need to hear about that product on an AMP page in their Chrome browser on their Android phone, probably following a Google search or noticing a story on Google news.
NateEag 6y ago
The new competitor would have to be so much better than Google that any given user has a >10% chance of switching, for the competitor to capture a remotely relevant marketshare. That seems fairly unlikely to me. Google gives you what you're looking for immediately for the average query, and anything beyond that is a power user detail not relevant to most people.

andai 6y ago

I've been thinking about this for a while. It's infeasible for any one person to run a Google-scale web crawler -- but if millions of people join in? I've used file sharing software which has the same "tree style" query forwarding. I think latency would be the big issue though, we're used to getting a result instantly -- and the right result too (Google does a lot of extra work knowing what you actually mean, not just keyword search).

mark_l_watson 6y ago
Common Crawl is a good resource if having crawl data potentially being from a month ago is OK for your application. Donate and use their crawl data.

alangibson 6y ago

I'm working in an architecture for a semi-decentralized, some-of-the-web crawler and search engine to try to address some of the problems with Google. Basically affinity groups maintain lists of links (sort of like social bookmarking) that are the crawl root urls. Crawling comes from Common Crawl plus updates for recency. Groups can pool resources to run their own indexes and receive a share of the profits from text display ads.

lhnz 6y ago
Is there anything that I can follow? I'm very interested in this -- I was considering making something similar myself.
- alangibson 6y ago
  I'll give you a shout when I publish the first draft. From your profile you seem like a pretty easy guy to reach. :)

avasthe 6y ago

Distributed schemes don't take off. Especially this one can lead to severe privacy / security loopholes. And what about people who rarely come online / have limited bandwidth? A more realistic dream would be finding good algorithms for "webscale" search engines, that all competitors (Bing, DDG) can use such that any one of them can't have excessive advantage over the others. Secondly, human curated directories may still have future. If I search Google for interesting links or just programming blogs on specific topic, I find lot of SEO optimized low quality content instead of deep technical content so that I often have to search through HN, reddit etc.. and sort by popularity - but searching for particular topic is difficult.