4 ms·

When Pagerank was created, it worked really well to find the most relevant sites. The old search engines such as AltaVista and Excite were drowning in spam sinc

by nikcub 16y ago

When Pagerank was created, it worked really well to find the most relevant sites. The old search engines such as AltaVista and Excite were drowning in spam since the SEO experts had figured them out. Does anybody else feel that Google has been figured out now, and that they are now just applying patches to a system that is fundamentally broken (in the same way Excite and AV did)? There might be an opportunity here for a new search engine - one that creates a new core method of ranking, as PageRank was, to filter out duplicate content, content farms, malware sites, etc. "Relevance = links" today looks as tired as "relevance = keyword density" did 10 years ago, but I have no idea what the new "relevance = " is

arkitaip 16y ago
Google is the best company to figuring out this "new" search engine. It's vastly cheaper and easier to continue tweaking their current system than it would be for someone else to code a search engine from scratch. IMHO there exists no silver bullet approach to search, instead it's a on-going quest/war consisting of infinite trials and errors. Also, I am not sure that the spam problem is as bad as people claim it to be. I have looked at virtually all of the lists of content farms that have been published since Google release their Personal Blocklist for Google Search and it seems to be that (a) it's all very subjective and people rarely agree that a particular site is a content farm (b) and therefore the number of sites that people seem to agree are farms aren't more than a dozen.
tnorthcutt 16y ago
Does anybody else feel that Google has been figured out now, and that they are now just applying patches to a system that is fundamentally broken (in the same way Excite and AV did)? I don't. I think they do a much better job of actively improving their offering. No matter what the system is, someone will always try to game it. Even if the people running the system are very smart and try very hard, the smart gamers will (briefly) succeed. That's ok, though - it pushes the people running the system to work harder to improve it. It really is in Google's best interest (IMO) to return great search results, so I think they'll continue to improve what they do.
alanthonyc 16y ago
The "fundamentally new" system for defining relevance is the social graph. It's why Facebook and Twitter are so hot.
- CWuestefeld 16y ago
  That's partly true, but given the way that the social graph is used today, I don't think it can satisfy the entire need. The social graph is essentially a personal thing, not professional. Assuming that we see a need to partition our professional and personal lives (and that should be true, in order to protect both sides), then the professional side isn't going to be reflected well in the personal graph. So long as our networking tools don't recognize this split, they are going to be (relatively) starved for content that is strongly oriented towards the professional world. For example, I wouldn't expect my wife to be able to find deep information about Medicare reimbursement if we relied on mining social sites.
  - johnzabroski 16y ago
    For a lot of professional needs, the most valuable information is hidden in the deep web. For example, many MIS/DSS/ERP/CRM/TLA systems don't provide a lot of public information about how their systems work, so if you are trying to evaluate the market place, it would probably take you 3-6 months of research. For example, I work at a healthcare services start-up, and we have one person full-time researching our competitors and figuring out how to differentiate us in the marketplace, as well as figure out what keywords someone might search for to find this kind of product. We're mainly mathematicians and smart programmers who found a way to break into the market initially, not marketers, so our terminology doesn't match domain experts well. Anecdotally, I think most users are VERY bad at ranking how valuable information is. For example, we'll get paid $100,000/year for something that takes 1 week and provides zero insight into their business, and then they will refuse to pay us at all for something that makes them $30,000,000 a year! This is so common that I am wondering if its more a symptom of human nature than our customers.
    - schwabacher 16y ago
johnzabroski 16y ago
There is a thing in HCI called the End-User Model Fallacy, where engineers try to create an interface based on a prototypical end-user, and then the real end-user spends time trying to figure out how they think the interface works, and it becomes a vicious cycle where the engineers think they are improving the interface, but in reality are just causing the real end-users to come up with a new model of the interface that they can use to get their work done. This Fallacy was first described to me in Brenda Laurel's Computers as Theatre. http://c2.com/cgi/wiki?BrendaLaurel http://c2.com/cgi/wiki?BrendaLaurel
iskander 16y ago
The original PageRank hasn't been used for a very long time-- the current system is a lot more sophisticated and blends many different ranking algorithms. I think that Google has done a good job of adapting to both the changes in web content and SEO scams. I do occasionally see noise in my search results, but there's always something useful among the top 3 links. This is quite different from the bad old days of Altavista, where you could get pages of outdated or irrelevant garbage.

by nikcub 16y ago

arkitaip 16y ago

Google is the best company to figuring out this "new" search engine. It's vastly cheaper and easier to continue tweaking their current system than it would be for someone else to code a search engine from scratch. IMHO there exists no silver bullet approach to search, instead it's a on-going quest/war consisting of infinite trials and errors. Also, I am not sure that the spam problem is as bad as people claim it to be. I have looked at virtually all of the lists of content farms that have been published since Google release their Personal Blocklist for Google Search and it seems to be that (a) it's all very subjective and people rarely agree that a particular site is a content farm (b) and therefore the number of sites that people seem to agree are farms aren't more than a dozen.

tnorthcutt 16y ago

Does anybody else feel that Google has been figured out now, and that they are now just applying patches to a system that is fundamentally broken (in the same way Excite and AV did)? I don't. I think they do a much better job of actively improving their offering. No matter what the system is, someone will always try to game it. Even if the people running the system are very smart and try very hard, the smart gamers will (briefly) succeed. That's ok, though - it pushes the people running the system to work harder to improve it. It really is in Google's best interest (IMO) to return great search results, so I think they'll continue to improve what they do.

alanthonyc 16y ago

The "fundamentally new" system for defining relevance is the social graph. It's why Facebook and Twitter are so hot.

CWuestefeld 16y ago
That's partly true, but given the way that the social graph is used today, I don't think it can satisfy the entire need. The social graph is essentially a personal thing, not professional. Assuming that we see a need to partition our professional and personal lives (and that should be true, in order to protect both sides), then the professional side isn't going to be reflected well in the personal graph. So long as our networking tools don't recognize this split, they are going to be (relatively) starved for content that is strongly oriented towards the professional world. For example, I wouldn't expect my wife to be able to find deep information about Medicare reimbursement if we relied on mining social sites.
- johnzabroski 16y ago
  For a lot of professional needs, the most valuable information is hidden in the deep web. For example, many MIS/DSS/ERP/CRM/TLA systems don't provide a lot of public information about how their systems work, so if you are trying to evaluate the market place, it would probably take you 3-6 months of research. For example, I work at a healthcare services start-up, and we have one person full-time researching our competitors and figuring out how to differentiate us in the marketplace, as well as figure out what keywords someone might search for to find this kind of product. We're mainly mathematicians and smart programmers who found a way to break into the market initially, not marketers, so our terminology doesn't match domain experts well. Anecdotally, I think most users are VERY bad at ranking how valuable information is. For example, we'll get paid $100,000/year for something that takes 1 week and provides zero insight into their business, and then they will refuse to pay us at all for something that makes them $30,000,000 a year! This is so common that I am wondering if its more a symptom of human nature than our customers.
  - schwabacher 16y ago

johnzabroski 16y ago

There is a thing in HCI called the End-User Model Fallacy, where engineers try to create an interface based on a prototypical end-user, and then the real end-user spends time trying to figure out how they think the interface works, and it becomes a vicious cycle where the engineers think they are improving the interface, but in reality are just causing the real end-users to come up with a new model of the interface that they can use to get their work done. This Fallacy was first described to me in Brenda Laurel's Computers as Theatre. http://c2.com/cgi/wiki?BrendaLaurel http://c2.com/cgi/wiki?BrendaLaurel

iskander 16y ago

The original PageRank hasn't been used for a very long time-- the current system is a lot more sophisticated and blends many different ranking algorithms. I think that Google has done a good job of adapting to both the changes in web content and SEO scams. I do occasionally see noise in my search results, but there's always something useful among the top 3 links. This is quite different from the bad old days of Altavista, where you could get pages of outdated or irrelevant garbage.