10 ms·

> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before. This makes i

> So, that’s another opportunity for DuckDuckGo, by ranking the results based on the content’s quality and relevance, as Google used to do before. This makes it sound like that's easy. Yes, SEO twattery is an issue, and with the growth of the web it's also much more prevalent and the stakes are higher. Google from 2022 has a much harder problem to solve than Google from 2000. DDG can't "just" solve that just as Google can't "just" solve that.

EVa5I7bHFq9mnYK 4y ago
It's humans (SEO) vs AI (Google). Humans win so far.
iopq 4y ago
Let users just rate that search result, so if someone else searched for the same thing and didn't find it relevant, they can thumbs down. If they found it relevant, they can thumb up.
- radlad 4y ago
  Would the votes apply only to the specific search query?
  - iopq 4y ago
    There should be a relevance score, and it might be cross-referenced to other similar search queries that mean the same thing, but that's a machine learning project all on itself
  - metadat 4y ago
    Site reputation score could be factored in.
- berkut 4y ago
  That only works if people are using the same search term AND looking for exactly the same thing... If they're looking for different search things but using the same search terms (i.e. some people are using sub-optimal search terms), that's just going to add effective random noise to the ratings. Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.
  - iopq 4y ago
    Nobody said I can't use a SIMILAR search query to have a slightly lower bound for relevance. Machine learning can identify if two queries mean a similar thing in the context of search
- wstrange 4y ago
  What makes you think a user rating system wouldn't be gamed?
  - iopq 4y ago
    You don't just average user ratings, you need to have a smart system. First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible. So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least). Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted. This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well. This system may not be perfect, but it would add value to a search engine
  - dazc 4y ago
    Especially since it has been tried and that is exactly what happened.
  - ricardo81 4y ago
    Exactly this. Pagerank is/was another form of voting.
- renewiltord 4y ago
  Mate, I'm going to pay $1000 and get a bunch of Filipinos in a LAN centre using that SaaS company that routes your requests through other people's VPN software to click a bunch of thumbs up. Watch as my cialis video tops your search for EC2 instances.
  - iopq 4y ago
    You need to do it with user accounts so the search engine team can remove fake clicks and ban the user. Users with a low amount of ratings won't have any say in the rating of the site, since they will have a low confidence. They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for? Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.
  - codegladiator 4y ago
    We already have that. At least I can hope for a variety when random users are also involved.
    - renewiltord 4y ago
      My Filipinos will win.
  - markles 4y ago
    Not OP, but allow users to follow others and create an aggregate filter. That leads to its own problems, but if results are done with a weighted preference depending on if have been noted as "good" by people I've followed, or to a lesser extent those they've followed, it may produce better results.
    - yunohn 4y ago
      I’m not sure the solution is to create a search results social network…
- glcheetham 4y ago
  Even better, let users leave comments on on search results. That stops all the "search query +reddit" searches because now Google is reddit. The discussions are happening on the search engine response page.
Nextgrid 4y ago
I disagree; Google isn't even trying - after all, Pinterest is still polluting image searches despite not even trying to do anything malicious. However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion. Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does. Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable. Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).
- bryanlarsen 4y ago
  99.9% of the web uses ads, analytics, affiliate links. I don't think anybody is interested in a search engine that only looks at 0.1% of the web.
  - deleted 4y ago
    [deleted]
  - badtension 4y ago
    This is the case because we optimized for it with a profit-centric internet.
  - Nextgrid 4y ago
    If search engines start penalizing this kind of content then the 0.1% might grow. Not to mention, most of my web search usage nowadays revolves around open-source library documentation, blogs, GitHub and occasionally StackOverflow, so being constrained to the 0.1% of the non-commercial web doesn't appear to be a downside at least when it comes to searching work-related issues as a programmer.
fbn79 4y ago
Yes. Why hadn't I thought about it? Everyone talks about how easy would be to solve this problem. Well if it so easy do it and become the next Page and Brin, instead of suggesting other to do.

EVa5I7bHFq9mnYK 4y ago

It's humans (SEO) vs AI (Google). Humans win so far.

iopq 4y ago

Let users just rate that search result, so if someone else searched for the same thing and didn't find it relevant, they can thumbs down. If they found it relevant, they can thumb up.

radlad 4y ago
Would the votes apply only to the specific search query?
- iopq 4y ago
  There should be a relevance score, and it might be cross-referenced to other similar search queries that mean the same thing, but that's a machine learning project all on itself
- metadat 4y ago
  Site reputation score could be factored in.
berkut 4y ago
That only works if people are using the same search term AND looking for exactly the same thing... If they're looking for different search things but using the same search terms (i.e. some people are using sub-optimal search terms), that's just going to add effective random noise to the ratings. Given how location-aware some things sometimes need to be (including spellings / alternative words, which in my experience, Google is a lot better at than DDG), I don't see how that works at all.
- iopq 4y ago
  Nobody said I can't use a SIMILAR search query to have a slightly lower bound for relevance. Machine learning can identify if two queries mean a similar thing in the context of search
wstrange 4y ago
What makes you think a user rating system wouldn't be gamed?
- iopq 4y ago
  You don't just average user ratings, you need to have a smart system. First, each user needs to have a history or they won't be considered in the rating almost at all. Then, each user's history gets analyzed for average rating vs. the average userbase so you can determine whether this user upvotes everything or only rates the sites that are terrible. So each user gets a score of "threshhold" of what rating site they will thumbs up. Some users will thumbs up a site that's 50% (mostly useless, but not outright spam), while some only upvote sites that are 80% (good content at the very least). Then regressed for this variable, you can calculate their feelings on each site and see if it correlates well with other users. The more ratings, the more each additional rating is weighted. This will reduce the number of ratings required for a statistically valid sample for each search query (since you need to tell users if it's relevant for the thing being searched, not just quality of the site). But you also need to fight bots as well. This system may not be perfect, but it would add value to a search engine
- dazc 4y ago
  Especially since it has been tried and that is exactly what happened.
- ricardo81 4y ago
  Exactly this. Pagerank is/was another form of voting.
renewiltord 4y ago
Mate, I'm going to pay $1000 and get a bunch of Filipinos in a LAN centre using that SaaS company that routes your requests through other people's VPN software to click a bunch of thumbs up. Watch as my cialis video tops your search for EC2 instances.
- iopq 4y ago
  You need to do it with user accounts so the search engine team can remove fake clicks and ban the user. Users with a low amount of ratings won't have any say in the rating of the site, since they will have a low confidence. They will first need to rate a few hundred sites (valid ratings, because if they don't have high validity with the rest of the sites in the database the confidence this is a valid user will be low) to give the spam site a high rating. So they will have already built a database of good ratings before they can give the last vote that they get paid for? Also, you can manually remove a spam site that uses this tactic and all their money will have been wasted.
- codegladiator 4y ago
  We already have that. At least I can hope for a variety when random users are also involved.
  - renewiltord 4y ago
    My Filipinos will win.
- markles 4y ago
  Not OP, but allow users to follow others and create an aggregate filter. That leads to its own problems, but if results are done with a weighted preference depending on if have been noted as "good" by people I've followed, or to a lesser extent those they've followed, it may produce better results.
  - yunohn 4y ago
    I’m not sure the solution is to create a search results social network…
glcheetham 4y ago
Even better, let users leave comments on on search results. That stops all the "search query +reddit" searches because now Google is reddit. The discussions are happening on the search engine response page.

Nextgrid 4y ago

I disagree; Google isn't even trying - after all, Pinterest is still polluting image searches despite not even trying to do anything malicious. However even if we assume that detecting spam content is hard, detecting the funding of said content is much easier - most of this content is there to get you to buy something or look at ads, and they use analytics tools to track conversion. Use the presence of ads, analytics, affiliate links and other marketing tools as a proxy for "spam" and downrank them, or offer the user a "non-commercial" option like Kagi does. Now you move the problem from old-school SEO "make my content rank higher" to "conceal my ads/analytics/etc from the search crawler so I don't get downranked". That is a very effective strategy because advertising has to be served via third-party code to protect against ad-fraud, but third-party scripts are trivially detectable. Now the spammers and their partners have a problem - the only way to fool the crawler is to self-host the ads, but doing that would expose the advertisers to ad fraud. You're pitting the spammers against each other in a game where the only winning move is not to play and the extra overhead will make the spam less profitable (or not profitable at all).

bryanlarsen 4y ago
99.9% of the web uses ads, analytics, affiliate links. I don't think anybody is interested in a search engine that only looks at 0.1% of the web.
- deleted 4y ago
  [deleted]
- badtension 4y ago
  This is the case because we optimized for it with a profit-centric internet.
- Nextgrid 4y ago
  If search engines start penalizing this kind of content then the 0.1% might grow. Not to mention, most of my web search usage nowadays revolves around open-source library documentation, blogs, GitHub and occasionally StackOverflow, so being constrained to the 0.1% of the non-commercial web doesn't appear to be a downside at least when it comes to searching work-related issues as a programmer.

fbn79 4y ago

Yes. Why hadn't I thought about it? Everyone talks about how easy would be to solve this problem. Well if it so easy do it and become the next Page and Brin, instead of suggesting other to do.