12 ms·

RIP Google PageRank score: A retrospective on how it ruined the web

282 points by adamcarson 11y ago

jandrese 11y ago
I feel like no matter how Google or any other search engine ranked pages, SEO firms would be there to game the system and make a mess of the web. Making Pagerank visible to people with one particular toolbar does seem like a fairly major misstep on Google's part. The majority of the people who would really care about that are the kind of people you shouldn't be encouraging. One of the obnoxious things about SEO is that if one person is doing it everybody has to do it. It's not necessarily enough to simply offer a better product at a better price. Luckily Google does try to reduce the effect of SEO. I notice for instance that StackExchange almost always beats out Expert Sex Change links these days.
- hire_charts 11y ago
  I'm not sure how much of it is that google tries to reduce the effects of SEO, or that they just keep moving the target. Every time they change their algorithm it seems like a bunch of sites who were practicing one strategy disappear, and before long a new strategy is discovered causing new sites to rise to the top. Ultimately Google's ability to quantify both quality and relevance will converge on actual quality and relevance. In other words, the way to "game" the system and practice SEO will be to create a site that is actually relevant with high-quality content. But we're not there yet, and who knows how close we'll actually get.
  - snewman 11y ago
    Obligatory: https://xkcd.com/810/ https://xkcd.com/810/
  - brosirmandude 11y ago
    This is definitely what is happening. I work in SEO and follow the industry closely, but the SEO industry itself is a giant echo-chamber. But essentially what it boils down to is: - make something your audience wants - make content your audience wants to share - "earn" links by creating useful resources & content Essentially, the way to get great rankings in Google is by actually having a website that people want to go to and is relevant to their interests. Trying to game the system is much harder these days, and honestly for the most part isn't worth the effort.
    - cookiecaper 11y ago
      This is what SEO firms like to say because it gives them cover and makes people think they're just content writers, but I don't think it's accurate. Backlinks are still king, even if your site has scarcely any content on it; not that it's bad to have more high-quality content, but it's not like Google weighs that strongly. A lot more human curation would be needed to determine that. You put "earn" in quotation marks, because these aren't organically earned except in a tiny fraction of cases. Most of them are planted.
    - gardano 11y ago
      As somebody with no dog in this fight, I cannot help but wonder if there's a correlation between PageRank and the rise of referring to text on a page as "Content", rather than, well, "An Article".
      - kolinko 11y ago
        Content is just a generic term. If you write a post or a Slideshare that has mostly pictures in it, it's still content but not an article.
        the_af 11y ago
        Not sure if the parent poster referred to this, but it's not absurd to consider that the term "content" may have harmful connotations. See Stallman's opinion on this (I know, an extremist view, but I think he has a point): "Words to avoid (or use with care)": http://www.gnu.org/philosophy/words-to-avoid.en.html#Content http://www.gnu.org/philosophy/words-to-avoid.en.html#Content > If you want to describe a feeling of comfort and satisfaction, by all means say you are “content,” but using the word as a noun to describe publications and works of authorship adopts an attitude you might rather avoid: it treats them as a commodity whose purpose is to fill a box and make money. In effect, it disparages the works themselves. If you don't agree with that attitude, you can call them “works” or “publications.” I think there may be a correlation between this negative phenomenon of gaming pagerank and referring to articles and pictures as "content", as if they were second-class citizens of the web.
  - cardine 11y ago
    I think that by the time that Google is able to quantify quality and relevance, people will be able to use AI to automatically create websites that are relevant and high quality. I'm not sure whether that is a good thing or not, but it would make sense that the sophistication of the AI used to game Google moves at about the same pace of Google's ability to stop people from gaming its algorithms.
    - TheOtherHobbes 11y ago
      I suspect it's not that hard to get a very rough assessment of quality and relevance - although it would only be useful with a giant multidimensional model of demographics and interests, because "relevant" isn't a scalar. My guess is Google doesn't do it because it costs too many cycles. Counting backlinks, supplemented by some very basic NLP, is very much cheaper and easier. The irony is that I wonder if people would pay for - or at least not mind - guided and prompted personal search if it produced highly relevant results. The sad truth is that the current sneaky lumping of broad demographic guessing with search history with backlink counting and a bit of NLP/timeline voodoo produces mediocre results for many kinds of searches. (I've just spent a very frustrating 15 minutes trying to find out if the raw datasets from KIC 8462852 are available online. Usually my search fu is pretty good, but I couldn't get a definitive answer.) I'm not sure to what extent Google's sales model relies on this. It's much easier to sell advertising if you don't offer an SLA, because it becomes the customer's fault if the service doesn't provide high quality results. A more effective service would increase ad buyer confidence and the price of niche ad sales would increase, but maybe not by enough to compensate for the loss of more generic ad sales overall.
  - scriptproof 11y ago
    There is no scientific definition of quality, so this can't happens (even if there is a easy ascertainment of garbage). Google is used to confuse quality and popularity, so maybe we will end with the more popular texts, ideas, beliefs, etc. on the top.
  - chei0aiV 11y ago
    People will still be able to game that scenario, just copy-paste from elsewhere.
  - deleted 11y ago
    [deleted]
- wilshiredetroit 11y ago
  PageRank wasnt bad. But the 'game-able' mechanism went on way too long.. and it was compounded when Google "went to bed" with SEO firms
  - hackuser 11y ago
    > it was compounded when Google "went to bed" with SEO firms What does this comment refer to?
    - wilshiredetroit 11y ago
      It just seems like Google was incentivised to have a relationship with SEO firms to make page rank work.
      - paulddraper 11y ago
        I doubt that. Google is Google because it was/is better than others at finding relevant information.
- cookiecaper 11y ago
  >It's not necessarily enough to simply offer a better product at a better price. It's not nearly enough to offer a better product at a better price. Our company, which was objectively superior to the bad imitators that followed us, was eviscerated by a competitor with a pedigree as a professional spammer -- his product rarely worked and we were constantly getting refugees once they were able to wade through the massive amounts of spam that promoted his solution and find some genuine information. It didn't help that he bought off niche webmasters to delete any mention of our service from their forums and only allow mention of his service. I have accepted it as a reality that "SEO consultants" (read: professional spammers) are a necessity if your site is going to get anywhere. Note we've always been SEO optimized, meaning our site was always search engine friendly and it contained a great deal more relevant content than our competitor's site. But because our competitor enlisted his network of spam websites to manufacture backlinks and actively sought to exclude mention of my product from the internet by paying off webmasters, he performed much better. People don't know and generally don't care if your product is better or worse. If you have a product that looks like it's functioning, that's all the development that needs to be done as long as it half-works 5-10% of the time. The rest of the money needs to go into spa--err, sorry, "internet marketing".
- GauntletWizard 11y ago
  Google is, like any ranking system, subject to Goodhart's law[1]. "When a measure becomes a target, it ceases to be a good measure." What has been Google's saving grace has been their skill at updating their target, attempting to turn the chaos and effort from SEO towards improvement in content. They have made an impossible problem into a mostly-solvable control systems[2] problem. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law [2] https://en.wikipedia.org/wiki/Control_system https://en.wikipedia.org/wiki/Control_system
- ChuckMcM 11y ago
  I see it slightly differently, the issue is money. When page rank was simply an internal relevance metric it was fine, but when page rank was the way to change how much revenue ads on your page brought you, well then it became much more than "relevance." Brad Templeton made the observation way early on that his humor mailing list went to over ten million people. If people sent him a penny if they laughed, and only 1% did, he would make $30,000 a month or $360,000 a year just for curating jokes. Scale + pennies can add up to big dollars on an individual level. PageRank, AdSense, AdWords, all fed this mechanism with a way to turn a small amount of work by an individual into disproportionate returns. It really has been no surprise that the web spam problem became much much worse than the email spam problem. There was a better built in payment mechanism.
- mistermann 11y ago
  > I notice for instance that StackExchange almost always beats out Expert Sex Change links these days. True, but they very often don't beat out numerous websites that have taken the open StackExchange content (or other similar, more specialized forums) and rehosted it on a new domain. This behavior should be easy to kill, Google should consider this blacklistable behavior and have an easy way to report any site doing it, with the number of StackExchange users out there these sites would be blacklisted within hours of being launched and before long peolple would simply stop doing it. But for some reason it is allowed to continue.
  - brongondwana 11y ago
    Consider this attack: 1) copy paste from from any random site to a StackExchange comment 2) report the original site for scraping StackExchange 3) lulz
    - zo1 11y ago
      And you think Google won't be able to compare timestamps of when each piece of content was inserted/updated?
    - mistermann 11y ago
      The StackExchange content is open, so I don't think the offending sites are doing anything wrong per se, excluding making the internet worse.
- seanp2k2 11y ago
  Also Quora. I luckily almost never see Quora results anymore. For a while, maybe a couple of years back, they'd rank on the first page for many technical queries. Clicking through would give you the question and a blocked answer. Really glad that Google decided that those businesses shouldn't win, because it's a really sleazy model.
rco8786 11y ago
Really crummy title. PageRank is the reason we HAVE a search as powerful as Google, and largely the reason the web is as good as it is today. Raise your hand if you want to go back to AltaVista/AskJeeves.
- SixSigma 11y ago
  To fully remember the evolution of search one needs to have been over 20 in 1995 and on the internet. That's probably quite a small number of people who are now over 20 and on the internet. Do you remember when Yahoo was a website about wrestling ? Or when having a good section in DMOZ was important ?
  - recursive 11y ago
    > Do you remember when Yahoo was a website about wrestling ? What? Really? I was around in 1995, and I don't recall that...
    - bryanlarsen 11y ago
      Jerry Wang had a webpage about wrestling that was hosted on the same server as "Jerry's Guide to the World Wide Web", the original name for Yahoo!.
    - kofejnik 11y ago
      i remember it being akebono.stanford.edu
  - klenwell 11y ago
    I don't remember Yahoo being a wrestling site. But I was over 20 in '95 and I can remember the internet before Google. It was a bit of a mess. Fun and amazing and kooky. But tantalizingly frustrating if you were trying to do any focused research. I can also remember the moment when I discovered Google. It was reading this article back in 2000 (I could have sworn it was 1998): http://www.newyorker.com/magazine/2000/05/29/search-and-deploy http://www.newyorker.com/magazine/2000/05/29/search-and-depl... If you want to travel back in time to the internet as it was when Google appeared, give it a read. For any gripes I may have about Google's search engine (like the fact I can never seem to easily relocate this article when I want to refer to it), it definitely solved more problems than it created.
_yosefk 11y ago
"How gravity ruined flying"? PageRank looking at links isn't some arbitrary thing, it's a source of information every good search will take into account.
tyingq 11y ago
The real problem is that Google was losing the link spam war until very, very recently. It was trivial to game them up until 2010, and only really became relatively difficult somewhere around 2012. And, the solution looks roughly like "weigh established authority to the point where it trumps relevance".
hartator 11y ago
I think it's odd to perceive the end of a relative transparent metric - whatever relevant or not it has been - as a good thing.
justinlardinois 11y ago
I never knew PageRank scores were visible, and I never used the web before Google. But this article is so far up its own ass. > Ever gotten a crappy email asking for links? Blame PageRank. Never mind that web rings were around long before Google and used the same tactics. > Ever had garbage comments with link drops? Blame PageRank. There are way more reasons spammers exist than just boosting PageRank. The author is acting like a) Google had less of an influence on the web before PageRank was public information and b) the web was somehow better both back then and before Google existed. There will always be people who want to game search engine results, regardless of how much information they know about their own standing, and the web was pretty much un-navigable pre-Google.
- hughw 11y ago
  > Never mind that web rings were around long before Google and used the same tactics. If you "never used the web before Google," how would you know this? I suppose you might have read about it. I did use the web before Google, and I don't remember web rings back then. There simply was no reason to do so, before Google started ranking pages based on the links. I do remember lots of BS meta keywords.
  - haberman 11y ago
    I used the web before Google too, and I remember them. Also, I do consider reading about something to be a valid way to learn about things you didn't directly witness, so it's strange that you would discount this. > There simply was no reason to do so, before Google started ranking pages based on the links. Webrings were useful for people who found a website interesting and wanted to visit other similar websites. The incentive to be in a webring was that you could get more exposure. Not to mention that the 90s were full of trendy things like this: "under construction" gifs, "valid HTML" buttons, etc. Web rings were one of those "clever" things you could add to a website.
    - james-skemp 11y ago
      Having had pages in Webrings way back when, it was definitely about exposure, but I think it was also an extension of groups (Yahoo! Groups being the biggest I can recall, but I think there was also OneGroup (?) - it maybe had purple in it's logo?). With Tripod, Geocities, Angelfire, and the like it was fairly easy to get a really basic page up, typically with a bunch of links to pages that you checked on regularly, and might be of interest to others. At least that's how it was for me. I think the rings I was part of included Terragen, POV-Ray, and Star Wars, and I can recall getting a couple people started with basic (and of course very ugly, with that same star field background for the SW-related pages) for people that I met in the various groups. Good times. EDIT: And don't forget the 'made with notepad' icons. Or recommending 800x600 or 1024x768 as the best resolution to view a site.
kazinator 11y ago
What difference does it make if the semantics of PageRank are still in place for determining position in the search index, but it is just hidden? You can still infer the approximate rank of a page by where it places relative to other pages, when searching for relevant keywords. Someone wanting to place ahead of the competition still has a function for measuring how well they are doing in SEO.
VikingCoder 11y ago
I just lost a ton of respect for Danny Sullivan. Every system can be gamed. Every system where money can be made WILL be gamed. It's a predator-prey relationship. The way this article was written made it sound like Google Search was a bane when it arrived. And sure, it was the worst Search Engine at the time, except for all the others that had been invented up until then.
- sullivandanny 11y ago
  You're reading things into this that I didn't write, I'd say. When Google arrived, it was a huge advance in search. It offered an obvious improvement in relevancy, which is why so many serious searchers switched to it from AltaVista and then users of other search engines moved over. Nothing I wrote suggested that Google was bad, didn't offer great relevancy or anything like that. My story is about what happened when Google revealed PageRank scores for pages across the web. That fueled an explosion in link buying and selling. It allowed people to attach Google's own score to a page, a value if you will that Google itself placed on those pages, which made it easier to then assign a monetary value. In turn, that lead to many of the woes that the web as a whole has to deal with today: understanding how to use nofollow to block links, to stay in Google's good graces. Spam mail pitching links, trying to buy links. Link spam I'm sure we'd have had some of this even without PageRank scores ever having been revealed. Perhaps it would have been as much, even. After all, it was well-known that Google was leveraging links as part of its ranking algorithm. The market would have been there. But I do think that releasing the PageRank scores accelerated market faster than it would have done otherwise. Back to the gaming -- again, it feels like you're reading stuff I didn't actually write. I'm certainly not saying that Google itself introduced the ability for people to try and game search engines. That was happening even before Google existed. Of course, Google initially thought it was immune. In 1998, Sergey Brin even said this on a panel that I moderated: "Google’s slightly different in that we never ban anybody, and we don’t really believe in spam in the sense that there’s no mechanism for removing people from our index. The fundamental concept we use is, you know, is this page relevant to the search? And, you know, some pages which, you know, they may almost never appear on the search results page because they’re just not that relevant." Google soon changed its view and introduced extensive spam fighting efforts. Those were inevitable. As you say, it was prey that would attract predators. And even with the link selling, it has done an admirable job fighting off the spam. It's not always perfect, but it's a very robust system. Nevertheless, the spam attempts will continue regardless if Google actually blunts them because, as the article explained, there's simply so many people with misconceptions that they'll chase anything anyway. PageRank scores fed into this, that's all.
al2o3cr 11y ago
Better title: "How SEO Asshattery Turned The Web To Shit"
rgovind 11y ago
Does anyone here think we need a search engine which lets us maintain large blacklists of websites. For example, if I am searching for information about airbnb, I do not want news websites like NY Times, WSJ, Forbes, Business standard etc to show up in the results at all. Any business related question on India is invariable dominated by Times of India and other newspapers. With google, its becoming increasingly difficult to filter out websites. Edit: Changed "on airbnb" to "about airbnb"
- legitster 11y ago
  Check out chrome extension "Personal Blocklist"
  - rgovind 11y ago
    Personal blocklist does not filter results at Google's algo level. It works at javascript level. So, its not really useful.
    - legitster 11y ago
      If I understand correctly, it takes the domains and includes them in an exclusion query on every search. So it does work on an algorithm level. For the use case you described it would work. You could also simply use the search operator site:airbnb.com
  - alxndr 11y ago
    I had used that extension for ages to prevent w3schools from appearing in my search results, but removed it when I eventually got tired of seeing search results jump around. (Because the extension works on a JS level, it needs to wait for the results page to load before it can strip out what you want to hide. Too often I'd see the results page begin drawing quickly, with a couple w3schools results at the top of the results, and my eyes would scan down to find the first non-w3schools result; then a second later, in the middle of finding the first such result, the w3schools results would be removed, and the non-w3schools result which I'd finally found and was about to click on has moved.)
kyledrake 11y ago
I first learned about this after starting https://neocities.org https://neocities.org and seeing a bunch of really garbage pages that were full of random text that linked to a derpie site somewhere. We get pagerank SEO spam from time to time, and it's pretty annoying. I have the tools to take care of it within 5 minutes every day, but I do worry that if we grow to a certain point it may no longer be possible for me to handle the problem alone. I'm sure many other sites have similar problems with comment spam, and I'd love to hear some advice on how to deal with this from sites that have the same problem. Right now our main lines of defense are a recaptcha (our last remaining third party embed, ironically sending user data to Google I'd rather not send to deal with a problem Google largely created), and a daily update of an IP blacklist we get from Stop Forum Spam. I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that.
- mmanfrin 11y ago
  This might be very heavy handed, but could you build something that added rel="nofollow" to all links on user pages until trust is verified?
  - softawre 11y ago
    > I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that. Did you read the comment?
  - kyledrake 11y ago
    I've pondered doing something like this, but it kindof crosses a line I'm not yet comfortable crossing (changing users' HTML without them noticing), and would require a way for me to convert from/back when the user edits their site. Technically it's pretty difficult: HTML/XML parsers like Nokogiri like to "fix" html when they parse it, so they can do a lot of changes to your document you didn't want to make. I've said I would decide to cross it if I ever ran into a major security issue, but so far that hasn't been the case. I've even decided against web page auto optimization ala things like ngx_pagespeed for this reason. One place I do cross this line is for our https://neocities.org/browse https://neocities.org/browse gallery in order to deal with iframe issues (I make the A links on a site open a new tab with a javascript inject), but that passive model won't work for pagerank issues because it uses the browser to make the changes. And honestly, let's be real, even if I put on nofollow and Google stops using pagerank, they're probably still going to do it because they're basically shady pagerank scam artists in the end, fooling gullible people into sending them money. They don't understand they're feeding into a botnet meets army of underpaid pagerank spammers, and it's basically impossible to fix this with education. It would be excellent if Google gave me an API to report pagerank spam. For all the money they've made on pagerank, it would be nice if they could defer some of that money into helping us deal with this, and would definitely help to improve their search results.
jzawodn 11y ago
Hah. Back in 2003 I wrote: "PageRank stopped working really well when people began to understand how PageRank worked. The act of Google trying to "understand" the web caused the web itself to change." http://jeremy.zawodny.com/blog/archives/000751.html http://jeremy.zawodny.com/blog/archives/000751.html It's amazing that it took this long.
- wpietri 11y ago
  You might be interested in Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law A friend mention that recently to me and it was really eye-opening. Once you start looking, the pattern is everywhere.
gchokov 11y ago
Indeed, one of the things I don't like (hate?) Google about is the SEO and PageRanking BS. All pages in the last 10 years are starting to look the same. All pages are becoming what Google wants them to be.
Animats 11y ago
Google has dealt with web spam by replacing it with their own ads. Search for "credit card" or "divorce lawyer". Everything above the fold is a Google ad. Air travel searches bring up Google's own travel info. No amount of SEO can compete with that. (I still offer Ad Limiter if you'd like to trim Google's in-house search result content down to a manageable level.)
- alphadevx 11y ago
  > Google has dealt with web spam by replacing it with their own ads. This. Google are not interesting in protecting us web users from SEO, but themselves (or more specifically their core ad business). After all why would you pay for Google ads if you could just get free traffic from Google via SEO techniques? So SEO is the logical competitor to Google.
  - johnward 11y ago
    I also believe this. Almost all updates to the search engine seem to be to drive SEOs towards adwords. It's not about protecting users, it's about profit.
kin 11y ago
Just because I can't see the score doesn't mean I'm not going to what I can to increase it.
runn1ng 11y ago
PageRank is still visible today?!? Where? (I am just curious, I thought it's not visible anywhere for years)
- tyingq 11y ago
  http://toolbarqueries.google.com/tbr?client=navclient-auto&ch=751310424703&features=Rank&q=info:http://news.ycombinator.com&num=100&filter=0 http://toolbarqueries.google.com/tbr?client=navclient-auto&c... The value of "ch" is a checksum you have to precalculate.
vorg 11y ago
Reading this makes one realize how easy it was for Groovy's Tiobe ranking to jump from #82 to #17 in just 12 months, as shown at http://www.tiobe.com/tiobe_index?page=Groovy http://www.tiobe.com/tiobe_index?page=Groovy , and the other spikes in its history.
- aikah 11y ago
  Are you saying that someone working on Groovy is doing "link optimisation" in order to get higher on Google thus Tiobe?
  - vorg 11y ago
    According to http://www.tiobe.com/tiobe_index?page=programminglanguages_definition http://www.tiobe.com/tiobe_index?page=programminglanguages_d... , Google is by far the largest component of Tiobe's calculation: Google.com: 7.69%, Google.co.in: 5.54%, Google.co.jp: 4.92%, Google.de: 3.69%, Google.co.uk: 3.08%, Google.com.br: 2.77%, Google.fr: 2.46%, Google.it: 2.15%, Google.es: 1.85%, Google.com.mx: 1.54%, Google.ca: 0.92%, Google.co.id: 0.62%. It seems the most likely reason for Groovy's strange behavior on Tiobe.
hackuser 11y ago
Another way to look at this is a blow to openness and a concentration of Google's power. The PageRank scores still exist, but they now will be known only by (some? all?) Google employees. Therefore, the data is no longer open and power is now more concentrated: Those who know someone at Google can find out their page rank score; the 99.999...% of the rest of the world cannot.

282 points by adamcarson 11y ago

I feel like no matter how Google or any other search engine ranked pages, SEO firms would be there to game the system and make a mess of the web. Making Pagerank visible to people with one particular toolbar does seem like a fairly major misstep on Google's part. The majority of the people who would really care about that are the kind of people you shouldn't be encouraging. One of the obnoxious things about SEO is that if one person is doing it everybody has to do it. It's not necessarily enough to simply offer a better product at a better price. Luckily Google does try to reduce the effect of SEO. I notice for instance that StackExchange almost always beats out Expert Sex Change links these days.

hire_charts 11y ago
I'm not sure how much of it is that google tries to reduce the effects of SEO, or that they just keep moving the target. Every time they change their algorithm it seems like a bunch of sites who were practicing one strategy disappear, and before long a new strategy is discovered causing new sites to rise to the top. Ultimately Google's ability to quantify both quality and relevance will converge on actual quality and relevance. In other words, the way to "game" the system and practice SEO will be to create a site that is actually relevant with high-quality content. But we're not there yet, and who knows how close we'll actually get.
- snewman 11y ago
  Obligatory: https://xkcd.com/810/ https://xkcd.com/810/
- brosirmandude 11y ago
  This is definitely what is happening. I work in SEO and follow the industry closely, but the SEO industry itself is a giant echo-chamber. But essentially what it boils down to is: - make something your audience wants - make content your audience wants to share - "earn" links by creating useful resources & content Essentially, the way to get great rankings in Google is by actually having a website that people want to go to and is relevant to their interests. Trying to game the system is much harder these days, and honestly for the most part isn't worth the effort.
  - cookiecaper 11y ago
    This is what SEO firms like to say because it gives them cover and makes people think they're just content writers, but I don't think it's accurate. Backlinks are still king, even if your site has scarcely any content on it; not that it's bad to have more high-quality content, but it's not like Google weighs that strongly. A lot more human curation would be needed to determine that. You put "earn" in quotation marks, because these aren't organically earned except in a tiny fraction of cases. Most of them are planted.
  - gardano 11y ago
    As somebody with no dog in this fight, I cannot help but wonder if there's a correlation between PageRank and the rise of referring to text on a page as "Content", rather than, well, "An Article".
    - kolinko 11y ago
      Content is just a generic term. If you write a post or a Slideshare that has mostly pictures in it, it's still content but not an article.
      - the_af 11y ago
        Not sure if the parent poster referred to this, but it's not absurd to consider that the term "content" may have harmful connotations. See Stallman's opinion on this (I know, an extremist view, but I think he has a point): "Words to avoid (or use with care)": http://www.gnu.org/philosophy/words-to-avoid.en.html#Content http://www.gnu.org/philosophy/words-to-avoid.en.html#Content > If you want to describe a feeling of comfort and satisfaction, by all means say you are “content,” but using the word as a noun to describe publications and works of authorship adopts an attitude you might rather avoid: it treats them as a commodity whose purpose is to fill a box and make money. In effect, it disparages the works themselves. If you don't agree with that attitude, you can call them “works” or “publications.” I think there may be a correlation between this negative phenomenon of gaming pagerank and referring to articles and pictures as "content", as if they were second-class citizens of the web.
- cardine 11y ago
  I think that by the time that Google is able to quantify quality and relevance, people will be able to use AI to automatically create websites that are relevant and high quality. I'm not sure whether that is a good thing or not, but it would make sense that the sophistication of the AI used to game Google moves at about the same pace of Google's ability to stop people from gaming its algorithms.
  - TheOtherHobbes 11y ago
    I suspect it's not that hard to get a very rough assessment of quality and relevance - although it would only be useful with a giant multidimensional model of demographics and interests, because "relevant" isn't a scalar. My guess is Google doesn't do it because it costs too many cycles. Counting backlinks, supplemented by some very basic NLP, is very much cheaper and easier. The irony is that I wonder if people would pay for - or at least not mind - guided and prompted personal search if it produced highly relevant results. The sad truth is that the current sneaky lumping of broad demographic guessing with search history with backlink counting and a bit of NLP/timeline voodoo produces mediocre results for many kinds of searches. (I've just spent a very frustrating 15 minutes trying to find out if the raw datasets from KIC 8462852 are available online. Usually my search fu is pretty good, but I couldn't get a definitive answer.) I'm not sure to what extent Google's sales model relies on this. It's much easier to sell advertising if you don't offer an SLA, because it becomes the customer's fault if the service doesn't provide high quality results. A more effective service would increase ad buyer confidence and the price of niche ad sales would increase, but maybe not by enough to compensate for the loss of more generic ad sales overall.
- scriptproof 11y ago
  There is no scientific definition of quality, so this can't happens (even if there is a easy ascertainment of garbage). Google is used to confuse quality and popularity, so maybe we will end with the more popular texts, ideas, beliefs, etc. on the top.
- chei0aiV 11y ago
  People will still be able to game that scenario, just copy-paste from elsewhere.
- deleted 11y ago
  [deleted]
wilshiredetroit 11y ago
PageRank wasnt bad. But the 'game-able' mechanism went on way too long.. and it was compounded when Google "went to bed" with SEO firms
- hackuser 11y ago
  > it was compounded when Google "went to bed" with SEO firms What does this comment refer to?
  - wilshiredetroit 11y ago
    It just seems like Google was incentivised to have a relationship with SEO firms to make page rank work.
    - paulddraper 11y ago
      I doubt that. Google is Google because it was/is better than others at finding relevant information.
cookiecaper 11y ago
>It's not necessarily enough to simply offer a better product at a better price. It's not nearly enough to offer a better product at a better price. Our company, which was objectively superior to the bad imitators that followed us, was eviscerated by a competitor with a pedigree as a professional spammer -- his product rarely worked and we were constantly getting refugees once they were able to wade through the massive amounts of spam that promoted his solution and find some genuine information. It didn't help that he bought off niche webmasters to delete any mention of our service from their forums and only allow mention of his service. I have accepted it as a reality that "SEO consultants" (read: professional spammers) are a necessity if your site is going to get anywhere. Note we've always been SEO optimized, meaning our site was always search engine friendly and it contained a great deal more relevant content than our competitor's site. But because our competitor enlisted his network of spam websites to manufacture backlinks and actively sought to exclude mention of my product from the internet by paying off webmasters, he performed much better. People don't know and generally don't care if your product is better or worse. If you have a product that looks like it's functioning, that's all the development that needs to be done as long as it half-works 5-10% of the time. The rest of the money needs to go into spa--err, sorry, "internet marketing".
GauntletWizard 11y ago
Google is, like any ranking system, subject to Goodhart's law[1]. "When a measure becomes a target, it ceases to be a good measure." What has been Google's saving grace has been their skill at updating their target, attempting to turn the chaos and effort from SEO towards improvement in content. They have made an impossible problem into a mostly-solvable control systems[2] problem. [1] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law [2] https://en.wikipedia.org/wiki/Control_system https://en.wikipedia.org/wiki/Control_system
ChuckMcM 11y ago
I see it slightly differently, the issue is money. When page rank was simply an internal relevance metric it was fine, but when page rank was the way to change how much revenue ads on your page brought you, well then it became much more than "relevance." Brad Templeton made the observation way early on that his humor mailing list went to over ten million people. If people sent him a penny if they laughed, and only 1% did, he would make $30,000 a month or $360,000 a year just for curating jokes. Scale + pennies can add up to big dollars on an individual level. PageRank, AdSense, AdWords, all fed this mechanism with a way to turn a small amount of work by an individual into disproportionate returns. It really has been no surprise that the web spam problem became much much worse than the email spam problem. There was a better built in payment mechanism.
mistermann 11y ago
> I notice for instance that StackExchange almost always beats out Expert Sex Change links these days. True, but they very often don't beat out numerous websites that have taken the open StackExchange content (or other similar, more specialized forums) and rehosted it on a new domain. This behavior should be easy to kill, Google should consider this blacklistable behavior and have an easy way to report any site doing it, with the number of StackExchange users out there these sites would be blacklisted within hours of being launched and before long peolple would simply stop doing it. But for some reason it is allowed to continue.
- brongondwana 11y ago
  Consider this attack: 1) copy paste from from any random site to a StackExchange comment 2) report the original site for scraping StackExchange 3) lulz
  - zo1 11y ago
    And you think Google won't be able to compare timestamps of when each piece of content was inserted/updated?
  - mistermann 11y ago
    The StackExchange content is open, so I don't think the offending sites are doing anything wrong per se, excluding making the internet worse.
seanp2k2 11y ago
Also Quora. I luckily almost never see Quora results anymore. For a while, maybe a couple of years back, they'd rank on the first page for many technical queries. Clicking through would give you the question and a blocked answer. Really glad that Google decided that those businesses shouldn't win, because it's a really sleazy model.

rco8786 11y ago

Really crummy title. PageRank is the reason we HAVE a search as powerful as Google, and largely the reason the web is as good as it is today. Raise your hand if you want to go back to AltaVista/AskJeeves.

SixSigma 11y ago
To fully remember the evolution of search one needs to have been over 20 in 1995 and on the internet. That's probably quite a small number of people who are now over 20 and on the internet. Do you remember when Yahoo was a website about wrestling ? Or when having a good section in DMOZ was important ?
- recursive 11y ago
  > Do you remember when Yahoo was a website about wrestling ? What? Really? I was around in 1995, and I don't recall that...
  - bryanlarsen 11y ago
    Jerry Wang had a webpage about wrestling that was hosted on the same server as "Jerry's Guide to the World Wide Web", the original name for Yahoo!.
  - kofejnik 11y ago
    i remember it being akebono.stanford.edu
- klenwell 11y ago
  I don't remember Yahoo being a wrestling site. But I was over 20 in '95 and I can remember the internet before Google. It was a bit of a mess. Fun and amazing and kooky. But tantalizingly frustrating if you were trying to do any focused research. I can also remember the moment when I discovered Google. It was reading this article back in 2000 (I could have sworn it was 1998): http://www.newyorker.com/magazine/2000/05/29/search-and-deploy http://www.newyorker.com/magazine/2000/05/29/search-and-depl... If you want to travel back in time to the internet as it was when Google appeared, give it a read. For any gripes I may have about Google's search engine (like the fact I can never seem to easily relocate this article when I want to refer to it), it definitely solved more problems than it created.

_yosefk 11y ago

"How gravity ruined flying"? PageRank looking at links isn't some arbitrary thing, it's a source of information every good search will take into account.

tyingq 11y ago

The real problem is that Google was losing the link spam war until very, very recently. It was trivial to game them up until 2010, and only really became relatively difficult somewhere around 2012. And, the solution looks roughly like "weigh established authority to the point where it trumps relevance".

hartator 11y ago

I think it's odd to perceive the end of a relative transparent metric - whatever relevant or not it has been - as a good thing.

justinlardinois 11y ago

I never knew PageRank scores were visible, and I never used the web before Google. But this article is so far up its own ass. > Ever gotten a crappy email asking for links? Blame PageRank. Never mind that web rings were around long before Google and used the same tactics. > Ever had garbage comments with link drops? Blame PageRank. There are way more reasons spammers exist than just boosting PageRank. The author is acting like a) Google had less of an influence on the web before PageRank was public information and b) the web was somehow better both back then and before Google existed. There will always be people who want to game search engine results, regardless of how much information they know about their own standing, and the web was pretty much un-navigable pre-Google.

hughw 11y ago
> Never mind that web rings were around long before Google and used the same tactics. If you "never used the web before Google," how would you know this? I suppose you might have read about it. I did use the web before Google, and I don't remember web rings back then. There simply was no reason to do so, before Google started ranking pages based on the links. I do remember lots of BS meta keywords.
- haberman 11y ago
  I used the web before Google too, and I remember them. Also, I do consider reading about something to be a valid way to learn about things you didn't directly witness, so it's strange that you would discount this. > There simply was no reason to do so, before Google started ranking pages based on the links. Webrings were useful for people who found a website interesting and wanted to visit other similar websites. The incentive to be in a webring was that you could get more exposure. Not to mention that the 90s were full of trendy things like this: "under construction" gifs, "valid HTML" buttons, etc. Web rings were one of those "clever" things you could add to a website.
  - james-skemp 11y ago
    Having had pages in Webrings way back when, it was definitely about exposure, but I think it was also an extension of groups (Yahoo! Groups being the biggest I can recall, but I think there was also OneGroup (?) - it maybe had purple in it's logo?). With Tripod, Geocities, Angelfire, and the like it was fairly easy to get a really basic page up, typically with a bunch of links to pages that you checked on regularly, and might be of interest to others. At least that's how it was for me. I think the rings I was part of included Terragen, POV-Ray, and Star Wars, and I can recall getting a couple people started with basic (and of course very ugly, with that same star field background for the SW-related pages) for people that I met in the various groups. Good times. EDIT: And don't forget the 'made with notepad' icons. Or recommending 800x600 or 1024x768 as the best resolution to view a site.

kazinator 11y ago

What difference does it make if the semantics of PageRank are still in place for determining position in the search index, but it is just hidden? You can still infer the approximate rank of a page by where it places relative to other pages, when searching for relevant keywords. Someone wanting to place ahead of the competition still has a function for measuring how well they are doing in SEO.

VikingCoder 11y ago

I just lost a ton of respect for Danny Sullivan. Every system can be gamed. Every system where money can be made WILL be gamed. It's a predator-prey relationship. The way this article was written made it sound like Google Search was a bane when it arrived. And sure, it was the worst Search Engine at the time, except for all the others that had been invented up until then.

sullivandanny 11y ago
You're reading things into this that I didn't write, I'd say. When Google arrived, it was a huge advance in search. It offered an obvious improvement in relevancy, which is why so many serious searchers switched to it from AltaVista and then users of other search engines moved over. Nothing I wrote suggested that Google was bad, didn't offer great relevancy or anything like that. My story is about what happened when Google revealed PageRank scores for pages across the web. That fueled an explosion in link buying and selling. It allowed people to attach Google's own score to a page, a value if you will that Google itself placed on those pages, which made it easier to then assign a monetary value. In turn, that lead to many of the woes that the web as a whole has to deal with today: understanding how to use nofollow to block links, to stay in Google's good graces. Spam mail pitching links, trying to buy links. Link spam I'm sure we'd have had some of this even without PageRank scores ever having been revealed. Perhaps it would have been as much, even. After all, it was well-known that Google was leveraging links as part of its ranking algorithm. The market would have been there. But I do think that releasing the PageRank scores accelerated market faster than it would have done otherwise. Back to the gaming -- again, it feels like you're reading stuff I didn't actually write. I'm certainly not saying that Google itself introduced the ability for people to try and game search engines. That was happening even before Google existed. Of course, Google initially thought it was immune. In 1998, Sergey Brin even said this on a panel that I moderated: "Google’s slightly different in that we never ban anybody, and we don’t really believe in spam in the sense that there’s no mechanism for removing people from our index. The fundamental concept we use is, you know, is this page relevant to the search? And, you know, some pages which, you know, they may almost never appear on the search results page because they’re just not that relevant." Google soon changed its view and introduced extensive spam fighting efforts. Those were inevitable. As you say, it was prey that would attract predators. And even with the link selling, it has done an admirable job fighting off the spam. It's not always perfect, but it's a very robust system. Nevertheless, the spam attempts will continue regardless if Google actually blunts them because, as the article explained, there's simply so many people with misconceptions that they'll chase anything anyway. PageRank scores fed into this, that's all.

al2o3cr 11y ago

Better title: "How SEO Asshattery Turned The Web To Shit"

rgovind 11y ago

Does anyone here think we need a search engine which lets us maintain large blacklists of websites. For example, if I am searching for information about airbnb, I do not want news websites like NY Times, WSJ, Forbes, Business standard etc to show up in the results at all. Any business related question on India is invariable dominated by Times of India and other newspapers. With google, its becoming increasingly difficult to filter out websites. Edit: Changed "on airbnb" to "about airbnb"

legitster 11y ago
Check out chrome extension "Personal Blocklist"
- rgovind 11y ago
  Personal blocklist does not filter results at Google's algo level. It works at javascript level. So, its not really useful.
  - legitster 11y ago
    If I understand correctly, it takes the domains and includes them in an exclusion query on every search. So it does work on an algorithm level. For the use case you described it would work. You could also simply use the search operator site:airbnb.com
- alxndr 11y ago
  I had used that extension for ages to prevent w3schools from appearing in my search results, but removed it when I eventually got tired of seeing search results jump around. (Because the extension works on a JS level, it needs to wait for the results page to load before it can strip out what you want to hide. Too often I'd see the results page begin drawing quickly, with a couple w3schools results at the top of the results, and my eyes would scan down to find the first non-w3schools result; then a second later, in the middle of finding the first such result, the w3schools results would be removed, and the non-w3schools result which I'd finally found and was about to click on has moved.)

kyledrake 11y ago

I first learned about this after starting https://neocities.org https://neocities.org and seeing a bunch of really garbage pages that were full of random text that linked to a derpie site somewhere. We get pagerank SEO spam from time to time, and it's pretty annoying. I have the tools to take care of it within 5 minutes every day, but I do worry that if we grow to a certain point it may no longer be possible for me to handle the problem alone. I'm sure many other sites have similar problems with comment spam, and I'd love to hear some advice on how to deal with this from sites that have the same problem. Right now our main lines of defense are a recaptcha (our last remaining third party embed, ironically sending user data to Google I'd rather not send to deal with a problem Google largely created), and a daily update of an IP blacklist we get from Stop Forum Spam. I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that.

mmanfrin 11y ago
This might be very heavy handed, but could you build something that added rel="nofollow" to all links on user pages until trust is verified?
- softawre 11y ago
  > I tried to do some Bayesian classification, but didn't make much progress unfortunately. And nofollow really isn't an option for me, as it would involve me manipulating other people's web sites and I don't want to do that. Did you read the comment?
- kyledrake 11y ago
  I've pondered doing something like this, but it kindof crosses a line I'm not yet comfortable crossing (changing users' HTML without them noticing), and would require a way for me to convert from/back when the user edits their site. Technically it's pretty difficult: HTML/XML parsers like Nokogiri like to "fix" html when they parse it, so they can do a lot of changes to your document you didn't want to make. I've said I would decide to cross it if I ever ran into a major security issue, but so far that hasn't been the case. I've even decided against web page auto optimization ala things like ngx_pagespeed for this reason. One place I do cross this line is for our https://neocities.org/browse https://neocities.org/browse gallery in order to deal with iframe issues (I make the A links on a site open a new tab with a javascript inject), but that passive model won't work for pagerank issues because it uses the browser to make the changes. And honestly, let's be real, even if I put on nofollow and Google stops using pagerank, they're probably still going to do it because they're basically shady pagerank scam artists in the end, fooling gullible people into sending them money. They don't understand they're feeding into a botnet meets army of underpaid pagerank spammers, and it's basically impossible to fix this with education. It would be excellent if Google gave me an API to report pagerank spam. For all the money they've made on pagerank, it would be nice if they could defer some of that money into helping us deal with this, and would definitely help to improve their search results.

jzawodn 11y ago

Hah. Back in 2003 I wrote: "PageRank stopped working really well when people began to understand how PageRank worked. The act of Google trying to "understand" the web caused the web itself to change." http://jeremy.zawodny.com/blog/archives/000751.html http://jeremy.zawodny.com/blog/archives/000751.html It's amazing that it took this long.

wpietri 11y ago
You might be interested in Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law A friend mention that recently to me and it was really eye-opening. Once you start looking, the pattern is everywhere.

gchokov 11y ago

Indeed, one of the things I don't like (hate?) Google about is the SEO and PageRanking BS. All pages in the last 10 years are starting to look the same. All pages are becoming what Google wants them to be.

Animats 11y ago

Google has dealt with web spam by replacing it with their own ads. Search for "credit card" or "divorce lawyer". Everything above the fold is a Google ad. Air travel searches bring up Google's own travel info. No amount of SEO can compete with that. (I still offer Ad Limiter if you'd like to trim Google's in-house search result content down to a manageable level.)

alphadevx 11y ago
> Google has dealt with web spam by replacing it with their own ads. This. Google are not interesting in protecting us web users from SEO, but themselves (or more specifically their core ad business). After all why would you pay for Google ads if you could just get free traffic from Google via SEO techniques? So SEO is the logical competitor to Google.
- johnward 11y ago
  I also believe this. Almost all updates to the search engine seem to be to drive SEOs towards adwords. It's not about protecting users, it's about profit.

kin 11y ago

Just because I can't see the score doesn't mean I'm not going to what I can to increase it.

runn1ng 11y ago

PageRank is still visible today?!? Where? (I am just curious, I thought it's not visible anywhere for years)

tyingq 11y ago
http://toolbarqueries.google.com/tbr?client=navclient-auto&ch=751310424703&features=Rank&q=info:http://news.ycombinator.com&num=100&filter=0 http://toolbarqueries.google.com/tbr?client=navclient-auto&c... The value of "ch" is a checksum you have to precalculate.

vorg 11y ago

Reading this makes one realize how easy it was for Groovy's Tiobe ranking to jump from #82 to #17 in just 12 months, as shown at http://www.tiobe.com/tiobe_index?page=Groovy http://www.tiobe.com/tiobe_index?page=Groovy , and the other spikes in its history.

aikah 11y ago
Are you saying that someone working on Groovy is doing "link optimisation" in order to get higher on Google thus Tiobe?
- vorg 11y ago
  According to http://www.tiobe.com/tiobe_index?page=programminglanguages_definition http://www.tiobe.com/tiobe_index?page=programminglanguages_d... , Google is by far the largest component of Tiobe's calculation: Google.com: 7.69%, Google.co.in: 5.54%, Google.co.jp: 4.92%, Google.de: 3.69%, Google.co.uk: 3.08%, Google.com.br: 2.77%, Google.fr: 2.46%, Google.it: 2.15%, Google.es: 1.85%, Google.com.mx: 1.54%, Google.ca: 0.92%, Google.co.id: 0.62%. It seems the most likely reason for Groovy's strange behavior on Tiobe.

hackuser 11y ago

Another way to look at this is a blow to openness and a concentration of Google's power. The PageRank scores still exist, but they now will be known only by (some? all?) Google employees. Therefore, the data is no longer open and power is now more concentrated: Those who know someone at Google can find out their page rank score; the 99.999...% of the rest of the world cannot.