4 ms·

It would be an interesting experiment if there was a search engine that ran on a heuristic of novelty rather than conformity. If you want to search for the hist

It would be an interesting experiment if there was a search engine that ran on a heuristic of novelty rather than conformity. If you want to search for the history of pineapples, you don't need 100 sites with the same info. The more varied info you get, the broader a picture you can paint. The downside is in the assumption that every difference is correct, but it's as dangerous as the assumption that only what is repeated is correct. Would be an interesting alternative at least.

dasil003 10y ago
The blackhat SEO would at least be interesting if not terrifying.
M_Grey 10y ago
Even better, add settings the user can tweak for which heuristic setup you want. Put it behind enough layers of dashboards that it doesn't terrify the average user.
apetresc 10y ago
Certainly it's not _as_ dangerous.
jtraffic 10y ago
You could also use current search algorithms but add a penalty for redundancy. Or perhaps group similar results. I've always like the idea of column-based web results. I suppose some of what Google does has this flavor.
jclos 10y ago
It's even more odd considering diversity in page ranking and novelty in recommendation are well studied (although not solved) problems in information retrieval, so it's not like there isn't a solution. And it's not like Google doesn't have the human or computer resources to tackle it.
Xeoncross 10y ago
I would assume that to counter the junk/spam/wrong information that would develop (even unintentionally) about a topic you would have to combine this with a "trust" ranking like PageRank. Basically, only more mainstream sites have the right to present different views which leads you back to square one: Repeating information.
manarth 10y ago
If you want to search for the history of pineapples, you don't need 100 sites with the same info. "Pineapples grow from a plant in the tropics." "Pineapples are grown in equatorial countries." --- "Pineapples, when ripe, have a very sweet taste." "Ripe pineapples have a high concentration of sugar." --- "The etymology of the word for pineapple is 'excellent fruit' in the language of Tupi, although the english English translation is an aberration, with a stem of 'pine' due to it's resemblance to a pine cone." "Most European languages name the fruit using the root word 'nanas', from the original native language where it was grown. Many languages have adapted the root word, although a few have a large degree of uncertainty, such as the origin of the reference to 'apple' in the English word." --- It would need a fairly decent AI to recognise the commonality of information in these examples.
- Taek 10y ago
  I think people often underestimate the power of much simpler methods. You'd probably get substantial improvements over the baseline just by penalizing sites for having the same words too many times. I forget the exact number, but I think it's something like 50% of the words in your average article appear only once. So if two articles have even 5% similarity among the words they only use once, they are probably quite similar. AI of course can get you much farther still, but you definitely don't need something that powerful for a convincing upgrade from baseline.
  - tomcam 10y ago
    Google is way way past that point. It worked that way 10 years ago and was exploited, so they added many more signals, generally far subtler than that.
zachkatz 10y ago
If you market it right, I think you could have a wildly successful "counterculture" search engine.
dredmorbius 10y ago
The heuristics you're looking for are accuracy and relevance. The problem is that PageRank is no longer delivering on these.

dasil003 10y ago

The blackhat SEO would at least be interesting if not terrifying.

M_Grey 10y ago

Even better, add settings the user can tweak for which heuristic setup you want. Put it behind enough layers of dashboards that it doesn't terrify the average user.

apetresc 10y ago

Certainly it's not _as_ dangerous.

jtraffic 10y ago

You could also use current search algorithms but add a penalty for redundancy. Or perhaps group similar results. I've always like the idea of column-based web results. I suppose some of what Google does has this flavor.

jclos 10y ago

It's even more odd considering diversity in page ranking and novelty in recommendation are well studied (although not solved) problems in information retrieval, so it's not like there isn't a solution. And it's not like Google doesn't have the human or computer resources to tackle it.

Xeoncross 10y ago

I would assume that to counter the junk/spam/wrong information that would develop (even unintentionally) about a topic you would have to combine this with a "trust" ranking like PageRank. Basically, only more mainstream sites have the right to present different views which leads you back to square one: Repeating information.

manarth 10y ago

If you want to search for the history of pineapples, you don't need 100 sites with the same info. "Pineapples grow from a plant in the tropics." "Pineapples are grown in equatorial countries." --- "Pineapples, when ripe, have a very sweet taste." "Ripe pineapples have a high concentration of sugar." --- "The etymology of the word for pineapple is 'excellent fruit' in the language of Tupi, although the english English translation is an aberration, with a stem of 'pine' due to it's resemblance to a pine cone." "Most European languages name the fruit using the root word 'nanas', from the original native language where it was grown. Many languages have adapted the root word, although a few have a large degree of uncertainty, such as the origin of the reference to 'apple' in the English word." --- It would need a fairly decent AI to recognise the commonality of information in these examples.

Taek 10y ago
I think people often underestimate the power of much simpler methods. You'd probably get substantial improvements over the baseline just by penalizing sites for having the same words too many times. I forget the exact number, but I think it's something like 50% of the words in your average article appear only once. So if two articles have even 5% similarity among the words they only use once, they are probably quite similar. AI of course can get you much farther still, but you definitely don't need something that powerful for a convincing upgrade from baseline.
- tomcam 10y ago
  Google is way way past that point. It worked that way 10 years ago and was exploited, so they added many more signals, generally far subtler than that.

zachkatz 10y ago

If you market it right, I think you could have a wildly successful "counterculture" search engine.

dredmorbius 10y ago

The heuristics you're looking for are accuracy and relevance. The problem is that PageRank is no longer delivering on these.