16 ms·

What we really need is a new Google, built on open principles (decentralized / peer to peer, fully free software, backed by a nonprofit), and focused on indexin

What we really need is a new Google, built on open principles (decentralized / peer to peer, fully free software, backed by a nonprofit), and focused on indexing the long tail of insightful content that is neglected by Google because it lacks SEO, popularity, links, and other metrics that Google find interesting but we don't necessarily do.

slfnflctd 6y ago
It's a fuzzy concept, but I think you're pointing toward something there's a need for. PageRank is/was useful in a lot of ways, it's just not enough by itself. Its weaknesses have been ever more apparent, and it has become less effective over time. There are so many possible viable methods for ranking search results! Particularly now with higher level textual analysis using AI/ML/[buzzword], and perhaps more importantly, the resurgence of interest in curated content. People are getting better at discerning curated-for-revenue vs. curated-with-love.
- the_other 6y ago
  > It's a fuzzy concept, but I think you're pointing toward something there's a need for. PageRank is/was useful in a lot of ways, it's just not enough by itself. Its weaknesses have been ever more apparent, and it has become less effective over time. Would you speak to why you think this way about PageRank? What are its shortcomings? To me, who only paid surface-level attention to this, it seemed like Google results were best when PageRank was the dominant metric. As they moved more and more in the direction of prioritising news, commerce and the aspects we call SEO, “number of links pointing TO the resource” became less and less important in the ranking. And as that happened, the quality of results dropped, and the content silo-ing rose up. PageRank was peer-review. SEO is “who shouts the loudest”.
  - Jtsummers 6y ago
    SEO with Google was about gaming PageRank. A lot of work happened in the 00s trying to prevent that, but it requires an army of moderators and others to identify good/bad links and link sources. Is HN a good source? Sure. Are HN comments a good source? Not once spammers realized they could drop links in comments to boost their site's results. How do you tell the difference? Plain PageRank is weak against this, and that's what motivated a lot of changes to the ranking system.
    - tonyarkles 6y ago
      Nofollow helps a fair bit but requires cooperation from all high ranking sites to ensure that they don’t leak their PageRank juice.
  - hedora 6y ago
    They had to abandon page rank because it was widely gamed. As far as I can tell, the main reason Google succeeded was that other search engines let advertisers buy placement for keywords (and didn’t label paid links). I heard from an industry insider that was able to strip the paid links that the engine they worked on gave results that were very similar to Google’s. The second big reason was that pagerank was a useful signal that hadn’t already been gamed to the point of uselessness. I think this let a tiny team blindside an entrenched industry. That’s not to say there’s no technical insight behind the page rank algorithm, but it was only a useful signal for a few years.
    - jabl 6y ago
      Before Google/pagerank took over, sites were successfully gaming the rankings of search engines like Altavista by adding every keyword they could think of in the html header, or hidden in the page with an invisible font etc. It got to the point Altavista became more or less useless, and when Google showed up on the market they quickly took it over. Seems the time is ripe for a new revolution. Doesn't have to be a better search engine, could be something completely different.
basch 6y ago
Agree on the long tail. And maybe a way to exclude bigger sites, and newer results (the time range search doesnt really work anymore.) I had a search today, and 7 of the top 10 results were from today. What I was looking for was NOT news, it was historical. If I wanted news, I would click the news tab. Having 7/10ths of the results come from today makes using google to search all of the web ever near useless, as todays noise is noisier than ever. I dont even care if they are defaults, but buttons to "exclude big sites" "exclude the news" or "exclude fresh results" would make search so much better.
- spc476 6y ago
  Here you go: https://millionshort.com/ https://millionshort.com/
  - dredmorbius 6y ago
    An equivalent which provides sources older than the past hour, day, week, month, year, decade, century, or millennium would be an opportunity here.
    - basch 6y ago
      Google has that, it's just stopped working. You'll end up with articles from 1997 that say they updated 4h ago. And the amount of seo garbage that gets through has polluted the remaining results. What would be really cool was if google could show what the results for a search looked like on a given day. Not the current algorithm, not any sites indexed since then, but what it looked like at the time. Going back to use 2010 Google would be a dream.
      - aspenmayer 6y ago
        Part of me wonders whether this is technically possible. I’m sure it’s an answerable question.
        6y ago
CJefferson 6y ago
The problem with any such system is there is money from capturing eyeballs, so once any system gets popular a lot of people will dedicate time to spamming it. I don't know how avoid that.
- Cilvic 6y ago
  Maybe there is a way to align spamming and curation? I mean what if all those people working on SEO would actually work on curation?
- akavel 6y ago
  Hm; a sibling comment about p2p made me start thinking about some kind of "authority" system, with personal tracking. I know it'd be a lot of work for everyone, bit it still sounds maybe doable to me. Like if everything posted to the "directories" was signed by its author, and you'd be able to mark entries (links) as valuable or not for yourself, which would be tracked in your local authors DB, by tweaking particular author's total score. Then you could share your list with friends, who could merge it into theirs, but still marked as coming from you. I know, kinda complex (and reminescent of gpg's "web of trust"), but maybe still could be an alternative to centralized services like google? Also the emerging p2p networks like dat/hypercore or ipfs or ssb already auto-sign people's published data, so maybe this could be tapped into...
  - bduerst 6y ago
    Even those types of authority systems can be gamed as well, unless you're also building your own recaptcha pattern analyzer too. The problem with decentralization is that it creates a power vaccuum that is filled by the most interested actors. Even bitcoin, with it's decentralization-by-design is actually centralized to a handful of miners in China.' If the goal is to rebuild something that is anything else other than profit based, you need to make sure the organization running it is strictly non-profit.
jandrese 6y ago
Of course the problem is that if your site becomes popular it will be overrun by SEOs pushing monetized vapid content. This is how people earn money, they aren't going to stop just because you want to return to a time before they ruined the internet.
- asdff 6y ago
  Results could be better filtered. SEO spam isn't very nuanced, it all reads the same and frequently plagiarized. I wonder if you can crudely use something like turnitin to filter for original content. Maybe you could try and make a model SEO article for your own search engine, just taking your existing SEO results and figuring out which parameters are contributing the most to their ranking, then filtering out results that contain these parameters that worked well in your model. Rinse and repeat as SEO writers try and step up their arms race, but they should always end up being foiled by your changes to the search engine after optimizing your own perfect SEO model regularly.
  - jandrese 6y ago
    SEO is exactly as sophisticated as it needs to be to get to the top of your rankings.
    - chongli 6y ago
      SEO sites have an Achilles' heel though: they need to make money somehow, and 99% it's going to be ads. So filter them out based on that. Create a space for a noncommercial web.
      - jefftk 6y ago
        They can also make money by selling things. Much harder to detect algorithmically than ads, and I don't know if you want to filter it out?
        dredmorbius 6y ago
StillBored 6y ago
For a long time I assumed that google indexed pretty much everything it, and it was only a question of providing a specific enough set of search terms to drag up older content. But what you hint at might be more correct these days. They are running a reverse wayback machine in that anything not changed in the last year gets removed. If you click the advanced search its "updated within" and the max timeframe is a year. In fact it seems the date range example doesn't even work: https://developers.google.com/custom-search/docs/structured_data#page_dates https://developers.google.com/custom-search/docs/structured_... If I fiddle with it, it returns a result, but I see an hit from just a few days ago at the top...
- koreth1 6y ago
  > They are running a reverse wayback machine in that anything not changed in the last year gets removed. Sometimes I wish that were true! Try Googling for, say, PostgreSQL documentation and the top result will often be for a 10-year-old version of the software.
  - pphysch 6y ago
    Nitpick, that's kind of a non example because the official Postgres docs let you swap versions more or less seamlessly. IME I will click the top result then click the correct version for my Postgres SE queries. 2 clicks, 0 scrolling.
    - koreth1 6y ago
      Sure, but 1 click and 0 scrolling would be better!
      - TuringTest 6y ago
        Try the button I'm feeling lucky!
        ZeikJT 6y ago
majormajor 6y ago
Any new google will become the current google because it will be gamed. I agree with the parent poster about hand-curated directories being the key to finding better stuff. It's necessarily much more distributed - because one person can't index everything - and therefore much harder to algorithmically game. Google has conditioned us into thinking that "an algorithm that automagically separates the wheat from the chaff" is the only way to do things. It worked for them for a while, but the adversarial forces of marketing, spam, malware, etc are very creative and fast-moving, and that's a lot for an algorithm to try to constantly reckon-with, so best case it'll probably stay a stalemate. But that's not the only way things can be.
- api 6y ago
  I was thinking the same thing. Search engines for the whole web are over. The concept doesn't stand up to spam of various kinds.
- esfandia 6y ago
  So we have hand-curated directories, and we have social networks. Put them together and you could have a scalable, crawlable, searchable, customizable and non-gamable decentralized indexing system. Just don't know why nobody has come with this yet. Maybe lack of (monetary) incentives? Maybe something for IETF or WWWC to initiate?
  - RNCTX 6y ago
    Social networks are also gamed. Since Twitter is their preferred platform, go put the activity of journalist Twitter accounts into a relational DB and start searching for who always boosts who. You'll find patterns. Of course there's nothing inherently wrong with this, but at the end of the day I don't need to know what a dozen NY Times journos think of a NY Times oped which is clearly written in bad faith, pushing a false narrative about a particular news event. Non-gameable ultimately means people who influence the results can have no monetary interest in the results. example: Twitter thread... https://twitter.com/jiatolentino/status/1263208982614814722 https://twitter.com/jiatolentino/status/1263208982614814722 ...vs reality... https://wearyourvoicemag.com/jia-tolentino-parents-teachers-harmed/ https://wearyourvoicemag.com/jia-tolentino-parents-teachers-...
raspyberr 6y ago
wiby.me can be useful for finding sites.
mrfusion 6y ago
If you have all the other stuff you might not really need peer to peer. Or maybe work in peer to peer after getting established? Also I like the idea of making search like Wikipedia where people can edit results. Obviously you’d need super genius level safeguards to protect against scammers but Wikipedia does it ok-ish.
WalterBright 6y ago
1. The internet is far too large to use human curation, an algorithm is needed. 2. An algorithm can always be gamed. You're stuck.
- indigochill 6y ago
  #1 doesn't matter because there's no rule that the entire internet needs to be visible. People only care about the sites they find value in. Search engines are one way to quickly locate those sites.
- rkagerer 6y ago
  Can you substantiate #1? The content on the sites I visit is created by humans. Until automation genuinely overtakes us, I'm not ready to accept at face value the scale of the internet has grown so large that humans couldn't tackle the problem.
  - WalterBright 6y ago
    Google says there are 2 billion web sites. Each one may consist of a huge number of web pages (see wikipedia.org). All I can say is, good luck with your human curation startup.
    - rkagerer 6y ago
      Sure, it also estimates more than twice that many people in the world with access to the internet. And Wikipedia is already well-curated by humans. If there were orders magnitude more pages than humans, I'd agree. But I'd also ask: Who created them all?
      - WalterBright 6y ago
        Wikipedia is one website. There are 2 billion. I don't believe wikipedia's methods scale to the internet. Also, any setup that allows everybody to be a moderator will be promptly gamed. What you want is what yahoo used to do - a hand-curated search engine. It worked when the internet was small, but got buried under the eventual avalanche of web sites.
jb3689 6y ago
> What we really need is a new Google Maybe, maybe not. How useful is Google (or search in general) to you? For me search is more of a convenience tool than for finding sites that have information. There are questions I need answered but without search would have an easy time figuring out (e.g. "how many cups are in a pint?"). Sometimes I want opinions, but I almost always am going to the same sources. Sometimes I use search because I'm too lazy to click around using a site's own search. The only things that are actually useful for me from search are for specific expert knowledge that I want in a structured manner (e.g. "what do I need to consider when buying a house?"), and those queries are incredibly few I feel like search is slowly becoming irrelevant
- ryan-allen 6y ago
  > Maybe, maybe not. How useful is Google (or search in general) to you? I think I use google in the same way as you. Most of the time, I could go to the websites directly (MDN, Stackoverflow, HN), but sometimes I'm trying to find something I don't know about, by trying different terms. I usually do this when I want a particular product but I don't know what it's called or what it is "smallest itx case without gpu", "midi router no power supply", "waterproof tarp diy tent setup". I installed a browser extension called uBlacklist as recommended by someone here a couple of weeks ago, so now 90% of the time Google search is like my Ctrl-P for MDN, Stackoverflow, etc since I've managed to filter out the sites I don't want to see results from.
ricardo81 6y ago
I don't think there needs to be a new Google or a requirement to be distributed. Simply having half a dozen or more search engines per country/language, with their own indexes and algorithms should help see the web more fully. ATM in English, Bing and Google have the largest indexes and Mojeek has its own index though smaller. DDG, Ecosia and others are just the Bing index re-ordered. I enjoyed the OP, though. And I think niche directories/blogrolls would be progress. The current centralised web is a result of everyone dancing to the tune/rules of the large platforms.
MaxBarraclough 6y ago
It's a pity the semantic web never took off. It might have greatly reduced the need for sophisticated centralised search-engines.
- fit2rule 6y ago
  I think the thing that happened, is the OS vendors spent too much time addicted to the web themselves, and forgot that one of the duties of the core OS of any device is to serve data from it, safely and securely, to any other device. Instead, most stacks are designed to enforce a walled garden, from which very little is shareable unless you go through the gateway (the web client app) to some approved destination. Semantic web and total availability at the personal-computer level, are aspects of OS design I wish had been paid more attention. Basically what we have now is a very, very expensive system of dumb terminals.
- greggman3 6y ago
  The semantic web will never work because there is no way to enforce it. If the topic "horses" become popular people will just start tagging their spam pages with "horse" to try to get people to look at their pages. As an example go look at the tags on soundcloud. People tag their songs with whatever they think will get people to take a listen.
  - MaxBarraclough 6y ago
    Dishonesty is a problem the web always faces, whether it's lying to a person or to a search-engine. It's not as if the semweb people weren't aware of the problem, and it's not self-evidently a fatal blow to the idea.
  - zozbot234 6y ago
    > The semantic web will never work because there is no way to enforce it. If the topic "horses" become popular people will just start tagging their spam pages with "horse" to try to get people to look at their pages. There are plenty of trusted sources that would not spam their pages in this way. And if you spam too much, you risk getting dropped from search indexes, directories, links etc. because your source is just not useful.
sacado2 6y ago
I'm surprised by this content, because Google actually changed the web for the better in this regard. Anybody remembers the ezinearticles era? Where people kept publishing the same short articles filled with just a few keywords and nothing more, over and over again, just to get backlinks? Back then you couldn't make a search on any topic without falling on page after page of those light articles filled with keywords only. Those article directories were eventually murdered by Google within the blink of an eye maybe a decade ago, and quite frankly on any given topic nowadays it's way easier to find good content rather than SEO filler. Google's algorithms will nowaydays favor fresh (aka published or updated recently), long content over short, "popular" (aka lots of links on it), duplicated content.
slightwinder 6y ago
> What we really need is a new Google, built on open principles (decentralized / peer to peer, fully free software, backed by a nonprofit), We have them, they suck. > and focused on indexing the long tail of insightful content that is neglected by Google because it lacks SEO How would you even define that? SEO is changing all the time. And google is fighting it all the time. And how would you prevent SEO focusing on that new searchengine? If it becomes big enough, people will optimize for it.
- Aeolun 6y ago
  I think the goal is to exclude anything that was created with a popular CMS (to exclude some content marketing), or in the top 100k websites. Then we might go back to something sort of interesting.
  - slightwinder 6y ago
    That is a good startingpoint, but also something SEO can very easy hack. Obfusicating the usual hints of a popular CMS is not hard. I wonder whether doing a parallel search on google and filter out by their top-results from your own results would be a feasiable solution? Add a filter on the top 500 websites, and whether known ad-sources are used and you might get slowly there. Maybe instead of a smart searchengine it would be better to focus on a dumb focus which gives access to all the metadata of a page too, and allows people to optimize for themself. Fulltext alone is not the only relevant content for good results. Google knows it and uses them, but has very limited acces to it for the enduser.
    - ajmurmann 6y ago
      I LOVE the idea of a search engine that lowers the rank of results with ads. This might be the key to success in finding projects of love rather than commercial click holes.
      - Aeolun 6y ago
        Oh. That’s also an idea, but maybe we’d still get tons of ad-less blogs that are just link farms to other sites. Still, I guess that’s only viable because Google rewards lots of links. If you just disable link relevance that part of gaming the system will be gone too.
pythonwizz 6y ago
So, basically Yacy? https://yacy.net/ https://yacy.net/

slfnflctd 6y ago

It's a fuzzy concept, but I think you're pointing toward something there's a need for. PageRank is/was useful in a lot of ways, it's just not enough by itself. Its weaknesses have been ever more apparent, and it has become less effective over time. There are so many possible viable methods for ranking search results! Particularly now with higher level textual analysis using AI/ML/[buzzword], and perhaps more importantly, the resurgence of interest in curated content. People are getting better at discerning curated-for-revenue vs. curated-with-love.

the_other 6y ago
> It's a fuzzy concept, but I think you're pointing toward something there's a need for. PageRank is/was useful in a lot of ways, it's just not enough by itself. Its weaknesses have been ever more apparent, and it has become less effective over time. Would you speak to why you think this way about PageRank? What are its shortcomings? To me, who only paid surface-level attention to this, it seemed like Google results were best when PageRank was the dominant metric. As they moved more and more in the direction of prioritising news, commerce and the aspects we call SEO, “number of links pointing TO the resource” became less and less important in the ranking. And as that happened, the quality of results dropped, and the content silo-ing rose up. PageRank was peer-review. SEO is “who shouts the loudest”.
- Jtsummers 6y ago
  SEO with Google was about gaming PageRank. A lot of work happened in the 00s trying to prevent that, but it requires an army of moderators and others to identify good/bad links and link sources. Is HN a good source? Sure. Are HN comments a good source? Not once spammers realized they could drop links in comments to boost their site's results. How do you tell the difference? Plain PageRank is weak against this, and that's what motivated a lot of changes to the ranking system.
  - tonyarkles 6y ago
    Nofollow helps a fair bit but requires cooperation from all high ranking sites to ensure that they don’t leak their PageRank juice.
- hedora 6y ago
  They had to abandon page rank because it was widely gamed. As far as I can tell, the main reason Google succeeded was that other search engines let advertisers buy placement for keywords (and didn’t label paid links). I heard from an industry insider that was able to strip the paid links that the engine they worked on gave results that were very similar to Google’s. The second big reason was that pagerank was a useful signal that hadn’t already been gamed to the point of uselessness. I think this let a tiny team blindside an entrenched industry. That’s not to say there’s no technical insight behind the page rank algorithm, but it was only a useful signal for a few years.
  - jabl 6y ago
    Before Google/pagerank took over, sites were successfully gaming the rankings of search engines like Altavista by adding every keyword they could think of in the html header, or hidden in the page with an invisible font etc. It got to the point Altavista became more or less useless, and when Google showed up on the market they quickly took it over. Seems the time is ripe for a new revolution. Doesn't have to be a better search engine, could be something completely different.

basch 6y ago

Agree on the long tail. And maybe a way to exclude bigger sites, and newer results (the time range search doesnt really work anymore.) I had a search today, and 7 of the top 10 results were from today. What I was looking for was NOT news, it was historical. If I wanted news, I would click the news tab. Having 7/10ths of the results come from today makes using google to search all of the web ever near useless, as todays noise is noisier than ever. I dont even care if they are defaults, but buttons to "exclude big sites" "exclude the news" or "exclude fresh results" would make search so much better.

spc476 6y ago
Here you go: https://millionshort.com/ https://millionshort.com/
- dredmorbius 6y ago
  An equivalent which provides sources older than the past hour, day, week, month, year, decade, century, or millennium would be an opportunity here.
  - basch 6y ago
    Google has that, it's just stopped working. You'll end up with articles from 1997 that say they updated 4h ago. And the amount of seo garbage that gets through has polluted the remaining results. What would be really cool was if google could show what the results for a search looked like on a given day. Not the current algorithm, not any sites indexed since then, but what it looked like at the time. Going back to use 2010 Google would be a dream.
    - aspenmayer 6y ago
      Part of me wonders whether this is technically possible. I’m sure it’s an answerable question.
      - 6y ago

CJefferson 6y ago

The problem with any such system is there is money from capturing eyeballs, so once any system gets popular a lot of people will dedicate time to spamming it. I don't know how avoid that.

Cilvic 6y ago
Maybe there is a way to align spamming and curation? I mean what if all those people working on SEO would actually work on curation?
akavel 6y ago
Hm; a sibling comment about p2p made me start thinking about some kind of "authority" system, with personal tracking. I know it'd be a lot of work for everyone, bit it still sounds maybe doable to me. Like if everything posted to the "directories" was signed by its author, and you'd be able to mark entries (links) as valuable or not for yourself, which would be tracked in your local authors DB, by tweaking particular author's total score. Then you could share your list with friends, who could merge it into theirs, but still marked as coming from you. I know, kinda complex (and reminescent of gpg's "web of trust"), but maybe still could be an alternative to centralized services like google? Also the emerging p2p networks like dat/hypercore or ipfs or ssb already auto-sign people's published data, so maybe this could be tapped into...
- bduerst 6y ago
  Even those types of authority systems can be gamed as well, unless you're also building your own recaptcha pattern analyzer too. The problem with decentralization is that it creates a power vaccuum that is filled by the most interested actors. Even bitcoin, with it's decentralization-by-design is actually centralized to a handful of miners in China.' If the goal is to rebuild something that is anything else other than profit based, you need to make sure the organization running it is strictly non-profit.

jandrese 6y ago

Of course the problem is that if your site becomes popular it will be overrun by SEOs pushing monetized vapid content. This is how people earn money, they aren't going to stop just because you want to return to a time before they ruined the internet.

asdff 6y ago
Results could be better filtered. SEO spam isn't very nuanced, it all reads the same and frequently plagiarized. I wonder if you can crudely use something like turnitin to filter for original content. Maybe you could try and make a model SEO article for your own search engine, just taking your existing SEO results and figuring out which parameters are contributing the most to their ranking, then filtering out results that contain these parameters that worked well in your model. Rinse and repeat as SEO writers try and step up their arms race, but they should always end up being foiled by your changes to the search engine after optimizing your own perfect SEO model regularly.
- jandrese 6y ago
  SEO is exactly as sophisticated as it needs to be to get to the top of your rankings.
  - chongli 6y ago
    SEO sites have an Achilles' heel though: they need to make money somehow, and 99% it's going to be ads. So filter them out based on that. Create a space for a noncommercial web.
    - jefftk 6y ago
      They can also make money by selling things. Much harder to detect algorithmically than ads, and I don't know if you want to filter it out?
      - dredmorbius 6y ago

StillBored 6y ago

For a long time I assumed that google indexed pretty much everything it, and it was only a question of providing a specific enough set of search terms to drag up older content. But what you hint at might be more correct these days. They are running a reverse wayback machine in that anything not changed in the last year gets removed. If you click the advanced search its "updated within" and the max timeframe is a year. In fact it seems the date range example doesn't even work: https://developers.google.com/custom-search/docs/structured_data#page_dates https://developers.google.com/custom-search/docs/structured_... If I fiddle with it, it returns a result, but I see an hit from just a few days ago at the top...

koreth1 6y ago
> They are running a reverse wayback machine in that anything not changed in the last year gets removed. Sometimes I wish that were true! Try Googling for, say, PostgreSQL documentation and the top result will often be for a 10-year-old version of the software.
- pphysch 6y ago
  Nitpick, that's kind of a non example because the official Postgres docs let you swap versions more or less seamlessly. IME I will click the top result then click the correct version for my Postgres SE queries. 2 clicks, 0 scrolling.
  - koreth1 6y ago
    Sure, but 1 click and 0 scrolling would be better!
    - TuringTest 6y ago
      Try the button I'm feeling lucky!
      - ZeikJT 6y ago

majormajor 6y ago

Any new google will become the current google because it will be gamed. I agree with the parent poster about hand-curated directories being the key to finding better stuff. It's necessarily much more distributed - because one person can't index everything - and therefore much harder to algorithmically game. Google has conditioned us into thinking that "an algorithm that automagically separates the wheat from the chaff" is the only way to do things. It worked for them for a while, but the adversarial forces of marketing, spam, malware, etc are very creative and fast-moving, and that's a lot for an algorithm to try to constantly reckon-with, so best case it'll probably stay a stalemate. But that's not the only way things can be.

api 6y ago
I was thinking the same thing. Search engines for the whole web are over. The concept doesn't stand up to spam of various kinds.
esfandia 6y ago
So we have hand-curated directories, and we have social networks. Put them together and you could have a scalable, crawlable, searchable, customizable and non-gamable decentralized indexing system. Just don't know why nobody has come with this yet. Maybe lack of (monetary) incentives? Maybe something for IETF or WWWC to initiate?
- RNCTX 6y ago
  Social networks are also gamed. Since Twitter is their preferred platform, go put the activity of journalist Twitter accounts into a relational DB and start searching for who always boosts who. You'll find patterns. Of course there's nothing inherently wrong with this, but at the end of the day I don't need to know what a dozen NY Times journos think of a NY Times oped which is clearly written in bad faith, pushing a false narrative about a particular news event. Non-gameable ultimately means people who influence the results can have no monetary interest in the results. example: Twitter thread... https://twitter.com/jiatolentino/status/1263208982614814722 https://twitter.com/jiatolentino/status/1263208982614814722 ...vs reality... https://wearyourvoicemag.com/jia-tolentino-parents-teachers-harmed/ https://wearyourvoicemag.com/jia-tolentino-parents-teachers-...

raspyberr 6y ago

wiby.me can be useful for finding sites.

mrfusion 6y ago

If you have all the other stuff you might not really need peer to peer. Or maybe work in peer to peer after getting established? Also I like the idea of making search like Wikipedia where people can edit results. Obviously you’d need super genius level safeguards to protect against scammers but Wikipedia does it ok-ish.

WalterBright 6y ago

1. The internet is far too large to use human curation, an algorithm is needed. 2. An algorithm can always be gamed. You're stuck.

indigochill 6y ago
#1 doesn't matter because there's no rule that the entire internet needs to be visible. People only care about the sites they find value in. Search engines are one way to quickly locate those sites.
rkagerer 6y ago
Can you substantiate #1? The content on the sites I visit is created by humans. Until automation genuinely overtakes us, I'm not ready to accept at face value the scale of the internet has grown so large that humans couldn't tackle the problem.
- WalterBright 6y ago
  Google says there are 2 billion web sites. Each one may consist of a huge number of web pages (see wikipedia.org). All I can say is, good luck with your human curation startup.
  - rkagerer 6y ago
    Sure, it also estimates more than twice that many people in the world with access to the internet. And Wikipedia is already well-curated by humans. If there were orders magnitude more pages than humans, I'd agree. But I'd also ask: Who created them all?
    - WalterBright 6y ago
      Wikipedia is one website. There are 2 billion. I don't believe wikipedia's methods scale to the internet. Also, any setup that allows everybody to be a moderator will be promptly gamed. What you want is what yahoo used to do - a hand-curated search engine. It worked when the internet was small, but got buried under the eventual avalanche of web sites.

jb3689 6y ago

> What we really need is a new Google Maybe, maybe not. How useful is Google (or search in general) to you? For me search is more of a convenience tool than for finding sites that have information. There are questions I need answered but without search would have an easy time figuring out (e.g. "how many cups are in a pint?"). Sometimes I want opinions, but I almost always am going to the same sources. Sometimes I use search because I'm too lazy to click around using a site's own search. The only things that are actually useful for me from search are for specific expert knowledge that I want in a structured manner (e.g. "what do I need to consider when buying a house?"), and those queries are incredibly few I feel like search is slowly becoming irrelevant

ryan-allen 6y ago
> Maybe, maybe not. How useful is Google (or search in general) to you? I think I use google in the same way as you. Most of the time, I could go to the websites directly (MDN, Stackoverflow, HN), but sometimes I'm trying to find something I don't know about, by trying different terms. I usually do this when I want a particular product but I don't know what it's called or what it is "smallest itx case without gpu", "midi router no power supply", "waterproof tarp diy tent setup". I installed a browser extension called uBlacklist as recommended by someone here a couple of weeks ago, so now 90% of the time Google search is like my Ctrl-P for MDN, Stackoverflow, etc since I've managed to filter out the sites I don't want to see results from.

ricardo81 6y ago

I don't think there needs to be a new Google or a requirement to be distributed. Simply having half a dozen or more search engines per country/language, with their own indexes and algorithms should help see the web more fully. ATM in English, Bing and Google have the largest indexes and Mojeek has its own index though smaller. DDG, Ecosia and others are just the Bing index re-ordered. I enjoyed the OP, though. And I think niche directories/blogrolls would be progress. The current centralised web is a result of everyone dancing to the tune/rules of the large platforms.

MaxBarraclough 6y ago

It's a pity the semantic web never took off. It might have greatly reduced the need for sophisticated centralised search-engines.

fit2rule 6y ago
I think the thing that happened, is the OS vendors spent too much time addicted to the web themselves, and forgot that one of the duties of the core OS of any device is to serve data from it, safely and securely, to any other device. Instead, most stacks are designed to enforce a walled garden, from which very little is shareable unless you go through the gateway (the web client app) to some approved destination. Semantic web and total availability at the personal-computer level, are aspects of OS design I wish had been paid more attention. Basically what we have now is a very, very expensive system of dumb terminals.
greggman3 6y ago
The semantic web will never work because there is no way to enforce it. If the topic "horses" become popular people will just start tagging their spam pages with "horse" to try to get people to look at their pages. As an example go look at the tags on soundcloud. People tag their songs with whatever they think will get people to take a listen.
- MaxBarraclough 6y ago
  Dishonesty is a problem the web always faces, whether it's lying to a person or to a search-engine. It's not as if the semweb people weren't aware of the problem, and it's not self-evidently a fatal blow to the idea.
- zozbot234 6y ago
  > The semantic web will never work because there is no way to enforce it. If the topic "horses" become popular people will just start tagging their spam pages with "horse" to try to get people to look at their pages. There are plenty of trusted sources that would not spam their pages in this way. And if you spam too much, you risk getting dropped from search indexes, directories, links etc. because your source is just not useful.

sacado2 6y ago

I'm surprised by this content, because Google actually changed the web for the better in this regard. Anybody remembers the ezinearticles era? Where people kept publishing the same short articles filled with just a few keywords and nothing more, over and over again, just to get backlinks? Back then you couldn't make a search on any topic without falling on page after page of those light articles filled with keywords only. Those article directories were eventually murdered by Google within the blink of an eye maybe a decade ago, and quite frankly on any given topic nowadays it's way easier to find good content rather than SEO filler. Google's algorithms will nowaydays favor fresh (aka published or updated recently), long content over short, "popular" (aka lots of links on it), duplicated content.

slightwinder 6y ago

> What we really need is a new Google, built on open principles (decentralized / peer to peer, fully free software, backed by a nonprofit), We have them, they suck. > and focused on indexing the long tail of insightful content that is neglected by Google because it lacks SEO How would you even define that? SEO is changing all the time. And google is fighting it all the time. And how would you prevent SEO focusing on that new searchengine? If it becomes big enough, people will optimize for it.

Aeolun 6y ago
I think the goal is to exclude anything that was created with a popular CMS (to exclude some content marketing), or in the top 100k websites. Then we might go back to something sort of interesting.
- slightwinder 6y ago
  That is a good startingpoint, but also something SEO can very easy hack. Obfusicating the usual hints of a popular CMS is not hard. I wonder whether doing a parallel search on google and filter out by their top-results from your own results would be a feasiable solution? Add a filter on the top 500 websites, and whether known ad-sources are used and you might get slowly there. Maybe instead of a smart searchengine it would be better to focus on a dumb focus which gives access to all the metadata of a page too, and allows people to optimize for themself. Fulltext alone is not the only relevant content for good results. Google knows it and uses them, but has very limited acces to it for the enduser.
  - ajmurmann 6y ago
    I LOVE the idea of a search engine that lowers the rank of results with ads. This might be the key to success in finding projects of love rather than commercial click holes.
    - Aeolun 6y ago
      Oh. That’s also an idea, but maybe we’d still get tons of ad-less blogs that are just link farms to other sites. Still, I guess that’s only viable because Google rewards lots of links. If you just disable link relevance that part of gaming the system will be gone too.

pythonwizz 6y ago

So, basically Yacy? https://yacy.net/ https://yacy.net/