4 ms·

I am actually wondering if search engines really need such a large index. The vast majority of sites in Google's index are crap. If you were able to better sele

I am actually wondering if search engines really need such a large index. The vast majority of sites in Google's index are crap. If you were able to better select which sites to index it might help search quality and improve efficiency.

nine_k 6y ago
How can you tell what is crap while crawling, is there enough time? How can you tell what is crap for the next person, and what is not crap? It's not an easy question.
- nwienert 6y ago
  People! Let the users help. All the best content sources on the internet are either experts or communities with voting / submissions.
  - sah2ed 6y ago
    > People! Let the users help. All the best content sources on the internet are either experts or communities with voting / submissions. There are over 1.7 billion websites [0], so the task of ranking content, the way algorithmic search engines do it in a matter of milliseconds, is not as easy as it sounds when you add humans into the mix. It would only end up the way Mahalo did [1]. [0] https://www.statista.com/chart/19058/how-many-websites-are-there/ https://www.statista.com/chart/19058/how-many-websites-are-t... [1] https://en.wikipedia.org/wiki/Mahalo.com https://en.wikipedia.org/wiki/Mahalo.com
    - nwienert 6y ago
      False dichotomy, you can use algorithms in tandem. And cherry picking... I remember Mahalo well, and one example of one version of it doesn’t prove anything. Mahalo is far from how I’d structure it. You can still automatically index but have users vote on the results. There are 1.7 billion websites and 3 billion+ users, and you don’t need that many to be active voters to help assist algorithms. Plus how many are at the top anyway? I’d love to downvote a ton of google results even if it only used it as a trainer for my own. Plus, there are so many “super curation” sites like here and Reddit that provide a big dataset curated by people automatically. Lean on them more. Everyone knows “site: reddit.com” or “site:stackoverflow.com” already give you better results. A simple upvote downvote on their results would let me downvote all the spam SEO sites. It wouldn’t take many votes for them to start tuning it. Stats are a good way to blind yourself. That algorithms scale doesn’t mean people don’t improve them. Google’s problem is they are too cocky about algorithms, but their algorithms fail compared the curated communities all over the web already.
  - asddubs 6y ago
    the other day I was searching "<sitename> feature" and from the results (which to be fair aren't many, since it was a fairly obscure site), once you get through the legit results on the site i was searching for, there were a bunch of markov generated spam things that had clearly scraped posts from various sites and tangled them together
BitwiseFool 6y ago
This is just a hunch but I imange the internet also follows the pareto principle and at least 80% of everything in google's index is basically worthless. We may end up with search engines that use multiple indices where each is curated to a certain domain of information, rather than a one-size-fits-all index.
a3camero 6y ago
I'm working on a search engine like this. So far just government websites but I think you're right about the need to focus on selected sites for some kinds of searches. For example, if you're looking for securities laws, you probably want to search government law sites and securities regulator sites. Maybe you want law blogs and law journal articles too, but you probably don't want Reddit or Hacker News.

nine_k 6y ago

How can you tell what is crap while crawling, is there enough time? How can you tell what is crap for the next person, and what is not crap? It's not an easy question.

nwienert 6y ago
People! Let the users help. All the best content sources on the internet are either experts or communities with voting / submissions.
- sah2ed 6y ago
  > People! Let the users help. All the best content sources on the internet are either experts or communities with voting / submissions. There are over 1.7 billion websites [0], so the task of ranking content, the way algorithmic search engines do it in a matter of milliseconds, is not as easy as it sounds when you add humans into the mix. It would only end up the way Mahalo did [1]. [0] https://www.statista.com/chart/19058/how-many-websites-are-there/ https://www.statista.com/chart/19058/how-many-websites-are-t... [1] https://en.wikipedia.org/wiki/Mahalo.com https://en.wikipedia.org/wiki/Mahalo.com
  - nwienert 6y ago
    False dichotomy, you can use algorithms in tandem. And cherry picking... I remember Mahalo well, and one example of one version of it doesn’t prove anything. Mahalo is far from how I’d structure it. You can still automatically index but have users vote on the results. There are 1.7 billion websites and 3 billion+ users, and you don’t need that many to be active voters to help assist algorithms. Plus how many are at the top anyway? I’d love to downvote a ton of google results even if it only used it as a trainer for my own. Plus, there are so many “super curation” sites like here and Reddit that provide a big dataset curated by people automatically. Lean on them more. Everyone knows “site: reddit.com” or “site:stackoverflow.com” already give you better results. A simple upvote downvote on their results would let me downvote all the spam SEO sites. It wouldn’t take many votes for them to start tuning it. Stats are a good way to blind yourself. That algorithms scale doesn’t mean people don’t improve them. Google’s problem is they are too cocky about algorithms, but their algorithms fail compared the curated communities all over the web already.
- asddubs 6y ago
  the other day I was searching "<sitename> feature" and from the results (which to be fair aren't many, since it was a fairly obscure site), once you get through the legit results on the site i was searching for, there were a bunch of markov generated spam things that had clearly scraped posts from various sites and tangled them together

BitwiseFool 6y ago

This is just a hunch but I imange the internet also follows the pareto principle and at least 80% of everything in google's index is basically worthless. We may end up with search engines that use multiple indices where each is curated to a certain domain of information, rather than a one-size-fits-all index.

a3camero 6y ago

I'm working on a search engine like this. So far just government websites but I think you're right about the need to focus on selected sites for some kinds of searches. For example, if you're looking for securities laws, you probably want to search government law sites and securities regulator sites. Maybe you want law blogs and law journal articles too, but you probably don't want Reddit or Hacker News.