14 ms·

Google algorithm change launched

688 points by Matt_Cutts 16y ago

Earlier this week Google launched an algorithmic change that will tend to rank scraper sites or sites with less original content lower. The net effect is that searchers are more likely to see the sites that wrote the original content. An example would be that stackoverflow.com will tend to rank higher than sites that just reuse stackoverflow.com's content. Note that the algorithmic change isn't specific to stackoverflow.com though. I know a few people here on HN had mentioned specific queries like [pass json body to spring mvc] or [aws s3 emr pig], and those look better to me now. I know that the people here all have their favorite programming-related query, so I wanted to ask if anyone notices a search where a site like efreedom ranks higher than SO now? Most of the searches I tried looked like they were returning SO at the appropriate times/slots now.

ironmanjakarta 16y ago
If a site ranks high in a SR, isn't it because it must have good backlinks? Shouldn't that be more important than whether it has original content or not? Maybe an aggregator displays the content in a more useful way than the originator so it gets linked to more than the originator. If the content creator doesn't like his content copied he can take it up with the copier. It's not Googles job to get involved in that. Google's job is to give the searcher a list of the sites that matches his keywords in an unbiased way. They should do that mostly on what the internet thinks is the best site, not what Google thinks is the best site.
joelhaus 16y ago
"scraper sites or sites with less original content" To clarify, this change measures unique content site-wide, not just for a particular web page/url? I may be too focused on semantics, but it would be important if you are trying to maximize visibility for a single page in Google's search results and your other pages have a significant amount of duplicate content. For instance, one page on your site is about a WordPress plugin you've created (it's totally unique), but on most of your other pages, you've copied and organized relevant sections of the WordPress Codex so that your users can easily find the documentation needed to customize the plugin. Is your unique webpage about the plugin safe? I was always under the impression that search rankings were determined on a page-by-page basis rather than site-wide. P.S. Sorry if this was already asked and I missed it.
rodh257 16y ago
So does this mean that Stack Overflow could remove the first tag from the start of the title and they would still rank above the scrapers? I found it really distracting and often double take on a SO result in google because I've scanned the first word and saw a spammy looking tag first.
pilooch 16y ago
One potential solution to spam and personalization of results lies in doing the job on your machine, much like you can get rid of spam in your local mailbox. I'd recommend 'Seeks', (http://www.seeks-project.info/ http://www.seeks-project.info/). It requires you run it on your machine, or use a public node. Though while on your machine, it 'learns' from your navigation and re-ranks the results based on these local data. Additionally I use regexps to remove websites I don't want to hear about, like expertexchange.com.
PHPAdam 16y ago
I run an letting agency (estate agency) I put the property's on my low ranking website. The content is not scraped, but re-published to several property portals, some big names. Who'm gets penalised? The higher ranking, more frequently indexed property portals or my local website.
cd34 16y ago
http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=draggable+jquery+revert#hl=en&sugexp=ldymls&xhr=t&q=draggable+jquery+reset&cp=22&qe=ZHJhZ2dhYmxlIGpxdWVyeSByZXNldA&qesig=_boNwazhD7dikYEGk18Yzg&pkc=AFgZ2tkkEndGT60KCdrxZwmd61AqPYPe0cczjEDCb9-zT8cumR5bv3R-CJqxMHe3aN4KcPD7E9DzrvWLdTJCC6KUZqxNKKej3g&pf=p&sclient=psy&safe=off&source=hp&aq=f&aqi=&aql=&oq=draggable+jquery+reset&pbx=1&fp=fca90e9507624f80 http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3... for me, the 10th result is: www.questionhub.com/StackOverflow/3910933 - Cached I think it is safe to say they are scraping, and, while there are some SO answers, that particular answer does correctly interpret what my intention was. In fact, the SO page scraped shows up as the 12th entry.
jmikel 16y ago
Anyone else notice that even Matt's site ranks below other sites who have copied the content after this change? Here's a search for the 3rd paragraph of content in the original blog post: http://goo.gl/vVs8A http://goo.gl/vVs8A Some sites, including this one: http://www.boonebank.com/brc/SBR_template.cfm?Document=headlines.cfm&article=1106 http://www.boonebank.com/brc/SBR_template.cfm?Document=headl... ... which links back to the original post (is that not supposed to acknowladge the original source of the article?) and quotes Matt ranks higher in my SERP results
badwetter 16y ago
Bravo!
- Matt_Cutts 16y ago
  I'll be curious to collect programming-related queries where we're not returning Stack Overflow or some other site that we should. Computer science and programming queries are easy for engineers to assess and say "Ah, here's something we need to do better on."
  - dminor 16y ago
    What's the best way to get these to you (after this article has dropped off the front page)?
    - Matt_Cutts 16y ago
      I'll still circle back to this page for quite a while, or you can tweet them to me. If it's long feedback, you can blog it and tweet a link to the blog post.
      - Natsu 16y ago
        As the guy in charge of fighting webspam, maybe this will help you find or improve upon some algorithmic means of detecting when lots of unrelated wikis start to get exploited to feed up advertisements. http://www.bay12forums.com/smf/index.php?topic=73045.0 http://www.bay12forums.com/smf/index.php?topic=73045.0 Apparently, for a short time (until the admins fixed it), a wiki for a game I play was hijacked to spam Google. Actually, said wiki (magmawiki.com) is under attack right now from spam user accounts. I don't run the place, but I thought I'd pass the information along because it might give you something to examine and because I have to imagine that anything that helps Google identify spam-hijacked wikis would be helpful to everyone.
      - cryptoz 16y ago
        This comment made me chuckle. I agree all this should be discussed out in the open, but blogging and then tweeting links to the blog post....what happened to email!? :)
flawawa2 16y ago
"We" is Google, could someone add that to the title? I was really confused what kind of algorithm would scan submitted pages at HN and rank based on that... On topic: Thanks! I would have thought Google already did that. Don't you also weigh in the age of the content? So if you have seen something earlier somewhere else it would be the "master" and rank higher?
alexsherrick 16y ago
thanks matt this is awesome!
- mebigfatguy 16y ago
  now if i could only get java 1.6 javadoc to show up, instead of java 1.4.2 javadoc, when i type in any arbitrary java class name.
bhavin 16y ago
I would like to point out one interesting thing I noticed today. I was looking for "gcc optimization flags for xeon". Following query is from google.com and contains no efreedom on the front page. http://www.google.com/search?hl=en&q=gcc+optimization+flags+for+xeon&aq=f&aqi=g-v1&aql=t&oq= http://www.google.com/search?hl=en&q=gcc+optimization+fl... Now, the same query from google.ie (ireland site) contains 2 efreedom on the top page! http://www.google.ie/search?hl=en&q=gcc+optimization+flags+for+xeon&btnG=Search&aq=f&aqi=&aql=&oq= http://www.google.ie/search?hl=en&q=gcc+optimization+fla... Why this strange search behavior to a query which has no relevance to user's location? P.S. I was logged on to my google account while searching, not sure if that has any effect whatsoever.
- bhavin 16y ago
  UPDATE: I am even more surprised to see that when I logged out of my account and searched again on google.ie, one of the efreedom disappeared!! So, does this mean that Google search now takes into account my past search result clicks (or rather mis-clicks)? Or ranks contents in someway that proves efreedom is somehow more relevant to me?
  - Matt_Cutts 16y ago
    If you're using personalized search, we do look at past clicks to change your ranking. So if you clicked on efreedom in the past, that could affect your ranking. You can add "&pws=0" (or use incognito mode in Chrome without logging in) to turn off personalized search and see whether that's the factor for you.
    - andybak 16y ago
      If I'm clicking around to crappy sites to figure out why they rank highly I don't really want them to get even higher in my personal search ranking! Is deleting them from search history enough to correct this effect?
    - bhavin 16y ago
      Adding &pws=0 turned the personalized search off and hence got rid of the efreedom. Also, I cleared my web history and searched again. The results are same now signed in or signed out. Problem solved! :) Thanks!
rgrieselhuber 16y ago
This is great news, thanks very much.
bretthellman 16y ago
Fantastic though I'd rather see google block sites like efreedom all together.
- jjclarkson 16y ago
  I think that would be a dangerous precedent to set. Whose opinion should decide what sites add value to content that may not be completely original? I'm not saying efreedom is not "being evil", but I could foresee someone somewhere using another's original content and making it more accessible. For example someone could easily improve the accessibility of the content from experts-exchange.com.
  - benologist 16y ago
    Google already makes that decision about lots of search results, usage on their services etc.
RealGeek 16y ago
Will this change effect sites like filestube.com and freshwap.net? FilesTube ranks for majority of the long tail keywords, even those not related to downloads/torrents/rapidshare. I see filestube's auto-generated search listing pages ranking on Google all the time. Pages like: http://www.filestube.com/m/matt+cutts http://www.filestube.com/m/matt+cutts http://www.filestube.com/g/google+scraper http://www.filestube.com/g/google+scraper Same goes for freshwap: http://www.freshwap.net/387/dl/Google+Matt+Cutts http://www.freshwap.net/387/dl/Google+Matt+Cutts These sites will give out an auto-generated page for every keyword you enter into it. Apparently, Google loves to index them... there are 126 million pages of files tube indexed in Google. I thought indexing search listing pages of other search engine was against Google's policies.
- xenophanes 16y ago
  There's some really annoying torrent sites like this. I mean, sites that pretend they have search results for whatever torrent you're searching for. Those show up on a google a lot and they're useless.
  - moultano 16y ago
    This is another class of problems we're working on. Expect some changes here in the next few months.
    - skinnymuch 16y ago
      Your account page doesn't say anything. I assume you work for Google?
      - moultano 16y ago
        Yep, in fact I wrote the change we're talking about in this thread. :)
        skinnymuch 16y ago
tristanperry 16y ago
Sounds good, thanks for the update Matt. I must ask though: I wonder whether you've seen the "January 26 2011 Traffic Change - Back to 'Zombie Traffic' " discussion over at Webmaster World? A number of webmasters there (who own websites with fully unique content) are reporting that they've seen lower quality content sites and/or content scrapers rank above their established sites with good content, starting from around the 26th Jan. There's no specific queries/websites talked about there (I might be wrong but I think Webmaster World has some rules preventing specific discussion of websites/queries), although I thought I'd flag it up since some webmasters have noticed adverse affects from a Jan 26th algo change; and it sounds like this might be the cause. Anywhoo, that being said: it's great to see Google continuing to be on the ball and responding to the recent feedback from various blogs and other sources (e.g. here).
- Matt_Cutts 16y ago
  I love Webmaster World, but one frustrating aspect of webmaster forums is that specific sites and queries are rarely given. It can be very hard to assess what's really going on since you don't know the specific site that's being discussed.
seanalltogether 16y ago
Matt, I just went through my search history because i remembered a very specific instance of seeing this. Here's the query. http://www.google.com/search?q=nstoolbar+bottom+bar&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a http://www.google.com/search?q=nstoolbar+bottom+bar&ie=u... You'll notice that efreedom.com shows on the first page with content taken directly from stackoverflow. While stackoverflow does show in the results, the exact page that efreedom copies does not. Anyway, I'm glad you guys are taking this seriously. For reference here is what I see right now - http://dl.dropbox.com/u/1437645/googlesearchresult.png http://dl.dropbox.com/u/1437645/googlesearchresult.png
- Confusion 16y ago
  It's strange how these results vary from person to person. When I click that link, I see an eFreedom link (with SO content) at #6 and a different SO link at #7.
- Matt_Cutts 16y ago
  It looks like we've got SO above efreedom for that query, but it's always nice to find a url that we didn't have that we'd like to be indexed. That lets us check whether we can improve our crawling/indexing. Thanks for the example!
  - millerc 16y ago
    If you tell me that sites who chronically scrape content elsewhere get a "pull-down weight" attached to the whole of their content, you've made my day.
    - bergie 16y ago
      That wouldn't be nice. There is a big difference between copying content from Stack Overflow, to running a Planet news aggregator on a community site for your open source project. The first is an ad scam, the second is valuable service to the members of the community.
  - chr15 16y ago
jacquesm 16y ago
Now that's good news. Thanks!
runjake 16y ago
FYI: Don't flag this because there's no link or "citation". Matt Cutts is the web spam guy at Google.
- sjs382 16y ago
  Here's the accompanying link: http://www.mattcutts.com/blog/algorithm-change-launched/ http://www.mattcutts.com/blog/algorithm-change-launched/
  - Matt_Cutts 16y ago
    Yup, I actually posted over here at HN a little bit before I did a post on my personal blog.
    - jjcm 16y ago
      I'd like to mention that this is really an impressive way of working with the community Matt. Companies say all the time that they value their customer's opinions, but rarely do you see a grievance that's posted on a social news site being a.) initially responded to by the guy who's responsible for it, and b.) amended by that person and his team with a request for review. It's heightened my faith that maybe Google can actually keep that small-company-feel no matter how large they get. I really hope you guys continue in this fashion, and thanks for the search fix.
      - yuhong 16y ago
        Yea, I have recently posted a comment starting with "Any Google employees here" knowing that there is more than one employee that posts here.
      - Matt_Cutts 16y ago
        HN has been a really high signal/noise site in discussing these issues, so it only seemed fair to give folks a heads-up here and see what other issues people were seeing. But thank you. :)
miah_ 16y ago
This is awesome! Now if only I could completely remove certain sites from my search results ala the Google Wiki stuff. I'd love to drop swik, and expertsexchange and a few other annoying sites. Its possible this algorithm change will make these sites less annoying to me though.
- SkyMarshal 16y ago
  Second that. WTB ability to customize my search results by blacklisting certain sites. Big change I know, but would be a hugely useful service Google could provide.
  - PaulJDrago 16y ago
    why not just use the - function? Create something like: http://www.google.com/search?q=google+-google.com&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a http://www.google.com/search?q=google+-google.com&ie=utf... I suspect you could hack together a quick greasemonkey script to automatically inject your negative site list when the URL is generated
    - miah_ 16y ago
      Because Chrome still doesn't sync your "search engine" settings. So maintaining a massive customized search engine across the multiple computers I use would be a pain. If this was a native feature it would work so long as I logged in to Google. Of course, if they fix chrome I'll be happy too.
    - Devilboy 16y ago
      This only works for a small number of blacklisted domains.
  - mythobit 16y ago
    I created a site that uses Google CSE to do just this. The link is: http://blacklist-search.appspot.com/ http://blacklist-search.appspot.com/
  - 16y ago
WillyF 16y ago
Matt, this is great news. How about sites that rank well with no content, just navigation? Here's an example: http://www.collegegrad.com/entryleveljob/entrylevelaccountingjobs.shtml http://www.collegegrad.com/entryleveljob/entrylevelaccountin... It's generally a high quality site, but that page has absolutely no relevance to the query except for a title tag and some internal anchor text. The search terms aren't even on the page. If I remember correctly, it used to rank #1 for "accounting entry level jobs," and now it's down to #8. My question is why is it even ranking at all? It's not even low quality content. It's no content.
- deleted 16y ago
  [deleted]
- tivaceous 16y ago
  I notice that you run a competing site to collegegrad.com - namely onedayonejob.com. Obviously you are likely to have paid attention to some of the things your competitor has been doing. But are you sure you're not just using this as an opportunity to stick it to them? If so, that would strike me as a distasteful use of this forum, especially since Matt has been very gracious to give this opportunity to the HN community.
  - wizard_2 16y ago
    Even if they were direct competitors with the exact same market and product (Which they don't appear to be), that could mean that he's got a much better idea of how his competitors may be doing spammy things then we do. Since these are algorithmic changes not site specific changes any resulting fix would get applied to his own site as well. The motives here don't offend me.
  - WillyF 16y ago
    I completely understand why you'd question my motives. I thought really hard about whether I should post this example or not. I tried to find an alternative example that would demonstrate the same problem, but this is the only one that I could come up with. I'm sure if I spent significantly more time looking I could have found something similar, but this example is very apparent to me because it's in SERPs that I watch closely. I'm not trying to "out" this site. I don't think they've done anything wrong or manipulative—they just have some pages with no content that are ranking well. I think that this type of issue should be on Matt's radar. Most of the site is of extremely high quality, and it deserves the high rankings that it gets. It's a page that is solely navigational structure, yet it ranks very well for a very specific keyword because of title tags and anchor text. We've already seen that Google has a problem (that they're working towards fixing) with low content from low and medium quality sites. What are they doing about low quality content (or non-content) from high quality sites?
spidaman 16y ago
Nice to hear about this change Matt. "Fuckin' efreedom..." is an oft heard missive around our office, godspeed in ranking them and their ilk down.
Aaronontheweb 16y ago
Glad to see that Matt went out of his way to help us geeks and the StackOverFlow community! In the future is there going to be some way for webmasters to do something like rel="canonical" across domains so if I want to syndicate a piece of content across two properties I own I can indicate which one is the original source? My understanding is that rel="canonical" is only meant to be used between pages on the same root domain today but I could be mistaken.
- sjs382 16y ago
  rel="canonical does work across domains.
  - Aaronontheweb 16y ago
    Good to know, thanks! I wasn't sure if that was the case or not. Is Google the only engine that supports it currently or is it standardized?
  - gregable 16y ago
    http://googlewebmastercentral.blogspot.com/2009/12/handling-legitimate-cross-domain.html http://googlewebmastercentral.blogspot.com/2009/12/handling-... was the official announcement.
- moultano 16y ago
  Syndicating content across your own domains is a common practice and will not be effected by this.
brown9-2 16y ago
Is this change restricted to programming-related queries only? I noticed today that a search for "mubarrek london" returns a page of results where every result on the first page, besides the top one, is spam from www.88searchengines.com, www.30searchengines.com, www.70searchengines.com, etc. I know this might not be related to topic of scraper sites directly, but not sure how else one can easily report these types of things.
- petervandijck 16y ago
  Google doesn't link to search engines in its results, that's their policy, so I would think those sites will be removed from their index at some point.
- treeface 16y ago
  I can't seem to replicate this: http://i.imgur.com/kDwPd.png http://i.imgur.com/kDwPd.png Are you sure there isn't something else going on?
  - brown9-2 16y ago
    Perhaps it's related to location, it's reproducible with incognito search (to make sure no personalized settings are being applied): http://i.imgur.com/Yd7FN.png http://i.imgur.com/Yd7FN.png
  - cd34 16y ago
    http://cd34.com/mubarek+london.png http://cd34.com/mubarek+london.png I get similar results as he does, with personalization, and, in Incognito mode.
  - tristanperry 16y ago
    I do see these problematic results: http://www.tristanperry.com/pics/SpammyResult.jpg http://www.tristanperry.com/pics/SpammyResult.jpg Searching from the UK, if that changes anything.
- boundlessdreamz 16y ago
  Tried the query and was able to replicate this https://skitch.com/techsutra/rm3gb/mubarrek-london-google-search https://skitch.com/techsutra/rm3gb/mubarrek-london-google-se...
giberson 16y ago
I imagine theres not much you can be specific about when talking about google's algorithm, but can you at least disclose if the identification of "original" content providers is "determined" (by some automatic process) or is it "specified" (manual intervention)? Stack overflow is an obvious benefactor of this new change, I'm just wondering if smaller content providers might benefit as well?
- moultano 16y ago
  There's nothing manual about this. Everybody who authors their own content should be helped. :)
  - Enlightenment 16y ago
    Please take care of the wikipedia scrapping trolls? http://meta.wikimedia.org/wiki/Live_mirrors http://meta.wikimedia.org/wiki/Live_mirrors http://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks http://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks http://openfacts.berlios.de/index-en.phtml?title=Copies_of_Wikipedia_content http://openfacts.berlios.de/index-en.phtml?title=Copies_of_W... http://en.wikipedia.org/wiki/Category:Websites_which_use_Wikipedia http://en.wikipedia.org/wiki/Category:Websites_which_use_Wik...
jjclarkson 16y ago
I ran across this list just now on sites that may be using SO content: http://meta.stackoverflow.com/questions/24611/is-it-legal-to-copy-stack-overflow-questions-and-answers/48962#48962 http://meta.stackoverflow.com/questions/24611/is-it-legal-to...
eurohacker 16y ago
may be here is a good place to ask - may be someone can answer - what does the term "original content" actually mean , original in what sense ? if you have a blog about wines, then what do you need to write - in order to be "original" blog about wines - does it mean mostly: 1) express original opinions about wines, 2) original structure of sentence and original wording - but the same opinions that 10 other sites write about, 3) original brands of wine you talk about - original names mentioned, 4) original content in the sense that you have not copied the text 100% from someone else - or what 5) combination of the factors mentioned above 6) something else that makes your content original ..
- socialmediaking 16y ago
  In order to be considered "original" from the search engine perspective, it needs to be around 30% new content. There are websites like http://www.copyscape.com/ http://www.copyscape.com/ that can tell you how original something appears. Internet marketers use software to "spin" text and make it appear unique. Sometimes the text is annoying for humans to read, but search engines eat it up. You can take one article and turn it into 50 with the click of a couple of buttons.
- lallysingh 16y ago
  I think this is a much simpler content, as in not being a literal copy of another's text. To be an "original" wine blog, write your own blog entries. Don't copy someone else's.
deleted 16y ago
[deleted]
hsmyers 16y ago
While I applaud such an improvement, I still despair over Google's inability to handle context. For instance the search phrase 'tex decorative rules' is totally miss managed. First it overrides the search and changes tex to text. Humorously enough, when you counter override back to tex, the search is even worse. We won't even speak of what happens when you change tex to latex--- porno doth ever rise to the top I guess. While we may be in a new millennium, some things remain far behind. And yes I know that things are being worked on from the those who were academic and are now working at Google, but still...
- Matt_Cutts 16y ago
  tex is a hard one. And [latex decorative rules] is even trickier--most people wouldn't expect technical documentation if they just saw that as a random query. That's more of an issue to pass on to the synonyms team and not in the scope of this change, but I'll pass this one on to the right folks
  - runningdogx 16y ago
    Would it be feasible to introduce a search customization setting: "lower the estimated probability that I mistyped"? I assume google uses something like two cut-offs: A < B, where if P(mistype)>A, it suggests but doesn't search for the correction, while if P(mistype)> B, it displays auto-corrected results. Simply an user option to nudge P(mistype) lower would probably suffice, or an option to increase B. A major usability problem for me with auto-corrected results is the two lines with links: "Showing results for <link>" and "Search instead for <link>". Interpreting those two lines takes me out of the flow of analyzing search results. I haven't yet been conditioned to click on the second link. I never want to click on the first link, because google is already showing me those results. I frequently search for unusual acronyms, terms, variables, made-up words, and gibberish that shows up in logs, among other things that google likes to "correct" for my benefit. I realize those use cases are not typical, but current auto-correction behavior can be extremely aggravating in those cases. Auto-suggest a correction all you want, but auto-correcting and suggesting the original is going too far I think, unless there are zero hits, or some equivalently strong algorithmic determination is made that I couldn't possibly want the query the way I typed it.
MelissaLB 16y ago
will there be any penalization for sites that use wikipedia content. Specifically a site that would NOT show up in the same search results as a wikipedia page but a commerce site that uses some unaltered wiki content in the descriptors of the products? I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th. Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?
MelissaLB 16y ago
will there be any penalization for sites that use wikipedia content. Specifically a site that would NOT show up in the same search results as a wikipedia page but a commerce site that uses some unaltered wiki content in the descriptors of the products? I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th. Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?
Alex3917 16y ago
Not exactly a scraper site, but if you do a search for "learn to hack" the top result is just a list of SEO keywords: http://www.learn-to-hack.com/ http://www.learn-to-hack.com/ Several of the other results are rather dubious as well. The reason I bring it up is because the Squidoo lens that comes up is something I made, and while certainly not perfect it's still a much better than many of the SEO spam sites and fake eBooks that rank above it. (And plus the ad revenue is going to charity rather than some shady organized crime ring.) Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works.
- moultano 16y ago
  >Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works. Definitely not a faux pas. Thanks for the example. My biggest annoyance in threads like these is people who write essays about their site losing traffic but then aren't willing to provide an url for people to check out.
  - Enlightenment 16y ago
    The guilty don't want google to know what they are doing <grin>. Keep up the good work squelching the trolls!
deleted 16y ago
[deleted]
angelbit 16y ago
Hi Matt. Why Google don't test the results after changes on algorithm? A simple program that store and compare n SERP results before and after the changes.
- Matt_Cutts 16y ago
  We do.
  - fzk390 16y ago
    I still see sites with low quality duplicate content but highly stuffed keywords. Here's one such example http://www.google.com/search?hl=en&q=internet+phone+service http://www.google.com/search?hl=en&q=internet+phone+serv... Check the listing for www.internetphoneguide.org Also for http://www.google.com/search?hl=en&q=voip+phone+service http://www.google.com/search?hl=en&q=voip+phone+service see the listing for www.zimbio.com which is not an authority on the topic. Last but not the least http://www.google.com/search?hl=en&q=home+phone+service http://www.google.com/search?hl=en&q=home+phone+service The site www.freelifelinephone.com jumped to Page 1 over night with the latest update.
hananc 16y ago
Please don't take my word for it. Hit http://duckduckgo.com http://duckduckgo.com with your programming queries. No - I am not affiliated and am not getting paid to write this.
- iPadDeveloper 16y ago
  Then why the advertisement? This is about Google's search result algorithms.
  - WilliamC 16y ago
    Because he couldn't find a better place to spam with traffic.
Nickwiz 16y ago
Hey matt, thanks for the update I wanted to know how would it work in this case: http://www.google.com/#q=major+online+dating+sites+koopa&hl=en&client=firefox-a&sa=N&rls=org.mozilla:en-US:official&fp=1 http://www.google.com/#q=major+online+dating+sites+koopa&#38... We submitted an article to ezinearticles from our blog.koopa.com. Just wondering how our blog is not remotely listed but ezine which we posted are article through our blog has? How does this work when our authors submit to article directories? On that same note I notice a bunch of other sites just below which have copies ezines article to the tee and are ranking higher then the our original blog that posted it. Is this related to the current algorithm change or just that our blog may not be indexed yet? Thanks for the update once again. I love the fact that you and the google team are constantly updating and changing your algo to give value to rightful content owners.
- moultano 16y ago
  What is the url of your blog?
  - Nickwiz2 16y ago
    http://blog.koopa.com http://blog.koopa.com
dcdan 16y ago
At what point does Google cross over from "ranked by algorithm" to "ranked by algorithm selection as editorialized by Googlers and bloggers?" Publicly discussing algorithms changes like this seems like a potential PR problem.
- Matt_Cutts 16y ago
  The whole history of Google is trying to find algorithms that encode our philosophy and mental model of what we think users want. We've been discussing algorithmic changes with people online since 2001, when GoogleGuy would show up on webmasterworld.com to dispel misconceptions.
  - dcdan 16y ago
    This algorithm change could come across as more editorial and less empirical. It's publicized as highly targeted ("slightly over 2% of queries"), responding to small tech discussion (Atwood, SO, HN, quality launch meeting), and seemingly rolled out in a short period of time. The media could easily boil this down to "a small number people felt sites they liked were under-ranked, so Google moved them up a week later." I mention this because Google often talks publicly about being entirely algorithmic, and elements of this narrative feel human.
    - nostrademons 16y ago
      2% of queries is huge. If you average that across a population (which of course isn't how it's actually distributed...), a searcher could expect to run across such a query every few days.
nikothefinn 16y ago
Since Matt is responding here, I figured this is worth a shot, no harm in asking. Matt, would love a response from you if you get a chance, since the Webmaster Tools appeals process gives no insight whatsoever to our situation. Following on from one of the comments here, namely the idea that "value is in the eye of the beholder", I'd like to raise our own plight. I run a number of aggregator sites - the largest and oldest of them being celebrifi.com, which was a PageRank 5 until Google de-indexed us in December (along with some of our other sites, but interestingly not all). A little background - the purpose of the sites is to aggregate, organize, rank and add context to what's happening in the news, with each site focusing a specific vertical. Think Techmeme, but with more context. I'll be the first to admit, there is no original content, but I strongly believe that we "add value" by figuring out what exactly is going on in any given story or blog post. We add value to publishers, by always linking to the original source (indeed, many publishers directly request that we add their feeds to the sources we track), we respect copyright by only displaying a short snippet of the original text and only displaying thumbnail images and we add value to users by giving them easy access to a lot more content on the same topic/story, all in the same place. Google's Quality Guidelines clearly state that duplicate content is penalized, and that is totally fine with us, but is it right to totally de-index a site for duplicate content? I wouldn't even want to rank above the original source for any given piece of content, as I respect the hard work that writers and publishers put into creating quality content, but aggregators who add value have a role to play in the content ecosystem. Digg, for example, uses the "wisdom of the crowd" to aggregate and rank content - hence adds value. Topix takes a local approach to aggregating content, and uses comments to rank content - hence adds value. We take a verticalized approach to aggregating and ranking content, and hence I believe that we add value. As mentioned above, we got de-indexed in December, and despite going through the appeals process, fixing a few things on our end to do with sitemaps, and clearing out some older "low quality" sources that we were tracking, we received no clarity into what our crime was. Matt, I'd like to raise this issue with you - both as it relates to us, but also as a general industry question - are all aggregators going to be de-indexed? And if not, which aggregators are and which aren't? What is the criteria, and who decides? If its algorithmic, then I am very curious to know what on our sites triggered the de-indexing? And even more curious to know why some of our sites got de-indexed, and some didn't. I have great respect for Google's efforts to clean up spam and low-quality content - and would always expect to see original content ranking higher than aggregated content. But to completely de-index an established aggregator site and strip it of its PageRank seems very draconian. I would love to hear your/Google's position, and look forward to some more clarity on both our situation, and the future of news/content aggregators. Respectfully yours, Niko
akie 16y ago
Yes - I've been doing a lot of YII related searches the past week, and I've noticed that a lot of times a site called 'devcomments.com' pops up somewhere in the first 5 results. Usually with a bogus page that does contain the keywords you were looking for, but not the actual discussion/forum thread. It appears they have copied their content directly from the official site, yiiframework.com. Example: http://www.google.com/search?q=Any+yii+way+to+get+the+previous+URL%3F http://www.google.com/search?q=Any+yii+way+to+get+the+previo... In that particular instance it is result number 3 (and the original is number 1), but on more than one occasion it was the top result and it's never what you are looking for.
samd 16y ago
I don't suppose you'll tell us how you know whether a site's content is original or not. What if you have a blog with lots of quotes from other sites; will that hurt your rankings because Google sees "unoriginal" content? Is there some ratio of original to unoriginal content that must be met to keep from being flagged as a scraper?
kqueue 16y ago
Here's a query that shows efreedom above SO. I did site:efreedom.com and picked few titles until I found one. query: Mailengine with .NET API The right page that should show up is http://stackoverflow.com/questions/1720900/mailengine-with-net-api http://stackoverflow.com/questions/1720900/mailengine-with-n... If you google the link below you'll see that the page is indexed. http://stackoverflow.com/questions/1720900/mailengine-with-net-api http://stackoverflow.com/questions/1720900/mailengine-with-n... However, the page doesn't show up when you use the query I mentioned.
seles 16y ago
This is good news, but I feel like it is only treating a symptom not the actual disease. If the algorithm properly detected site relevance, importance and viewer satisfaction, those copycat sites should never have ranked higher in the first place. In a way this is admitting that it is impossible to stop the gaming of search engine optimization, and that the only way to deal with it is to "win" in some special cases. That being said I provide no real solution, this is a huge problem with millions behind each side. Although this is good news it is also gloom news.
- LiveTheDream 16y ago
  How would an algorithm detect viewer satisfaction?
  - seles 16y ago
    How long user stays on page (hint would be how soon they make next search/click), whether or not they continue looking for something under a similar search query, having actual buttons users can click to rate, etc.
    - moultano 16y ago
      Spam gets upvoted to the top of reddit on a regular basis. Most people are pretty apathetic to legitimacy most of the time. All we ever have is proxies for quality, the hope is that we have enough of them to cover the space.
kunjaan 16y ago
I googled for my own name "Kunjan Kshetri" and I still see sites like 1. www.peekyou.com/_kshetri 2. www.yasni.com/kunjan+kshetri/check+people 3. mobile.aa.com/mt/net35.ccs.neu.edu/home/kunjan/?un_jtt in the front page. These sites reused my homepage or just have my name listed. They too add no original content. Please remove these sites. Please note that searching in Bing produces more original content results.
- brianpan 16y ago
  Google doesn't just remove sites from their index. They will always be there. But if you think a different site should rank higher than these spammy ones, you should mention it and maybe someone can look at it. (I don't work for Google.)
  - kunjaan 16y ago
    I meant remove them at least from my front page.
    - porges 16y ago
      For a while there was a Google feature that let you do that, along with up/downvoting results. It got removed though, and I've never read why :/
avstraliitski 16y ago
Very happy to see the end of copied Wikipedia sites.
klbarry 16y ago
Hi Matt, I'm sure your busy and don't want to answer all SEO related questions anyway. I wanted to ask, though, what Google things about the importance of exact anchor text in rankings as remarked upon in http://www.seomoz.org/blog/how-organized-crime-is-taking-control-of-googles-search-results http://www.seomoz.org/blog/how-organized-crime-is-taking-con...?
EGreg 16y ago
I got 99 comments but of which this ain't one :)
endergen 16y ago
Matt would you comment on the thought process of Google on curation. There are sites that are generally strongly disliked. Examples: Demand Media in general. Or say w3schoolS.com for JavaScript/CSS/HTML. I can understand that there would likely be a deluge of legal action against Google if this was done in a heavy handed way. Or is it a principled thing where everyone should be treated equal? If so isn't that a lot of algorithm ideology. In practice hard rules without human judgment leads to ridiculous edge cases or bypassing the intent of the rules. I always assumed you curated in a way via having teams that specialized in creating topic specific search techniques which then all get combined together via topic detection or some other meta algorithm. This would be a good balance between manual curation and having it be a scalable approach.
bkaid 16y ago
While you are at it, can you go ahead and de-index this site: http://www.google.com/search?q=site:livestrong.com http://www.google.com/search?q=site:livestrong.com
23david 16y ago
Will this new change impact the issue where scrapers that take videos and video descriptions from youtube and turn them into 'blog posts' show up higher than the youtube page that contains the actual original content? I worked for nearly a year producing a few hundred videos only to find that spam sites (usually running adsense ads) were showing up ahead of the video on youtube.com and my own site. Our site's domain name was even plastered all over the description and it still didn't matter. The spam sites still showed up way ahead of us in the search results.
tmsh 16y ago
One caveat, though I imagine this has been thought of before, is that mobile versions of sites often have the same content as full-browser versions of sites. So ideally, perhaps m.google.com would be able to sort through this and not penalize the duplicated-nature of the mobile version.... Anyway, something to think about if you haven't already.
jswinghammer 16y ago
Ok here is the search that first brought this issue to my attention: mysql spatial index example lft rgt I see the following order: 1: http://planet.mysql.com/entry/?id=23512 http://planet.mysql.com/entry/?id=23512 2: http://efreedom.com/Question/1-1743894/Mysql-Optimizing-Finding-Super-Node-Nested-Set-Tree http://efreedom.com/Question/1-1743894/Mysql-Optimizing-Find... 3: http://explainextended.com/2009/09/29/adjacency-list-vs-nested-sets-mysql/ http://explainextended.com/2009/09/29/adjacency-list-vs-nest... 4: http://stackoverflow.com/questions/1743894/mysql-optimizing-finding-super-node-in-nested-set-tree http://stackoverflow.com/questions/1743894/mysql-optimizing-...
deleted 16y ago
[deleted]
gbesen 16y ago
Hi Matt, I would like to ask if vertical search engines like vast.com is considered as a scraper site? Real estate search, used car search and other vertical search concepts do need traffic from google. I for one find it valuable to visit a vertical search engine that saves me from visiting site after site to find the car / house I want. Does this mean the end of vertical search engines? (I'm not affiliated with vast.com in any way) I'm sure many people would like to know about this, thank you.
Galaxis 16y ago
My current favourite spammy search (Windows process names) seems slightly improved, but still produces mostly unusable results. Try something like: http://www.google.com/search?q=hidfind.exe http://www.google.com/search?q=hidfind.exe Another similar search area (Windows drivers for various hardware) has improved a lot judging from the few examples I just tried - actual hardware manufacturers are now amongst the top results for a change. So that's quite a step forward...
- quag 16y ago
  Repeating a quick search I was doing a week or so ago to find drivers, I'm not seeing an improvement. Asus actually has a great driver site, that is way better and less hassle than the spam sites, but I've yet to see it show up in search results. I only found it because one of the spam sites was using asus's servers to serve up the drivers. http://www.google.co.nz/search?sourceid=chrome&ie=UTF-8&q=sis+asus+motherboard+drivers http://www.google.co.nz/search?sourceid=chrome&ie=UTF-8&...
kqueue 16y ago
The fact that efreedom results are showing up in the results is irritating by itself. We all know efreedom is spam, and so does Google. Now I am always on alert when clicking on a link in the results to avoid the spam pages like efreedom, expertexchange and whatnot. Why not just remove them from search results? They are causing enough bad publicity to Google.
forgotAgain 16y ago
When I search for "rabbitmq exchange declare" (no quotes) I noticed the following. Mailing list entries from http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-... show up from old.nabble.com before the original source. From a quick look there is significant improvement.
JWilder 16y ago
matt, kudos! this is great news. a question that I always pondered, what is googles approach on more clever forms of rehashed content that involve photoshopping / cropping an original image, is this something that google looks at? Example source photo: (http://shoes.n-sb.org/img/thumbs/472153d5fc674fa1c685f3c7814ecee9.jpg http://shoes.n-sb.org/img/thumbs/472153d5fc674fa1c685f3c7814...) rehashed image: (http://images.sneakernews.com/wp-content/uploads/2011/01/nike-sb-blazer-low-end-theory.jpg http://images.sneakernews.com/wp-content/uploads/2011/01/nik...)
cperciva 16y ago
searchers are more likely to see the sites that wrote the original content This is great news, but I have to wonder: How do you figure out which site wrote the original content? I'm wondering based on what happened to Tarsnap last week, where hackzq8search.appspot.com outranked tarsnap.com on a search for "tarsnap".
MiaBellezza 16y ago
http://www.homebasedesign.com/unusual-pergola-designs-deck-and-unusual-pergola-pictures-by-miabellezza/ http://www.homebasedesign.com/unusual-pergola-designs-deck-a... this site has copied numerous sites on the net and though Google adsense was taken away, it still ranks well. No page is their own. All are copies. The link above is referencing a search for unusual pergola designs, of which my lens is first in search. It has copied my whimsical garden art, etc. These are mine and the rest are copies of other people's sites. Though they never seem to update their garbled text and offend with blown up pictures, they still continue to rank well in searches.... Simply Amazing and this situation has been ongoing for a year. Yes I asked them several times to take out the pages to no avail. The whole site should be taken down. It's a joke. here is their copy of my Whimsical Garden Art: http://www.homebasedesign.com/whimsical-garden-art-ideas-for-your-whimsical-garden-by-miabellezza/ http://www.homebasedesign.com/whimsical-garden-art-ideas-for... my lens that they copied: http://www.squidoo.com/whimsical-gardens-art http://www.squidoo.com/whimsical-gardens-art There are other sites they created and did the very same thing: homedesigndecorating.com hometrendesign.com onhomedesign.com hahoy.com no original content, all copied. MiaBellezza

688 points by Matt_Cutts 16y ago

ironmanjakarta 16y ago

If a site ranks high in a SR, isn't it because it must have good backlinks? Shouldn't that be more important than whether it has original content or not? Maybe an aggregator displays the content in a more useful way than the originator so it gets linked to more than the originator. If the content creator doesn't like his content copied he can take it up with the copier. It's not Googles job to get involved in that. Google's job is to give the searcher a list of the sites that matches his keywords in an unbiased way. They should do that mostly on what the internet thinks is the best site, not what Google thinks is the best site.

joelhaus 16y ago

"scraper sites or sites with less original content" To clarify, this change measures unique content site-wide, not just for a particular web page/url? I may be too focused on semantics, but it would be important if you are trying to maximize visibility for a single page in Google's search results and your other pages have a significant amount of duplicate content. For instance, one page on your site is about a WordPress plugin you've created (it's totally unique), but on most of your other pages, you've copied and organized relevant sections of the WordPress Codex so that your users can easily find the documentation needed to customize the plugin. Is your unique webpage about the plugin safe? I was always under the impression that search rankings were determined on a page-by-page basis rather than site-wide. P.S. Sorry if this was already asked and I missed it.

rodh257 16y ago

So does this mean that Stack Overflow could remove the first tag from the start of the title and they would still rank above the scrapers? I found it really distracting and often double take on a SO result in google because I've scanned the first word and saw a spammy looking tag first.

pilooch 16y ago

One potential solution to spam and personalization of results lies in doing the job on your machine, much like you can get rid of spam in your local mailbox. I'd recommend 'Seeks', (http://www.seeks-project.info/ http://www.seeks-project.info/). It requires you run it on your machine, or use a public node. Though while on your machine, it 'learns' from your navigation and re-ranks the results based on these local data. Additionally I use regexps to remove websites I don't want to hear about, like expertexchange.com.

PHPAdam 16y ago

I run an letting agency (estate agency) I put the property's on my low ranking website. The content is not scraped, but re-published to several property portals, some big names. Who'm gets penalised? The higher ranking, more frequently indexed property portals or my local website.

cd34 16y ago

http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=draggable+jquery+revert#hl=en&sugexp=ldymls&xhr=t&q=draggable+jquery+reset&cp=22&qe=ZHJhZ2dhYmxlIGpxdWVyeSByZXNldA&qesig=_boNwazhD7dikYEGk18Yzg&pkc=AFgZ2tkkEndGT60KCdrxZwmd61AqPYPe0cczjEDCb9-zT8cumR5bv3R-CJqxMHe3aN4KcPD7E9DzrvWLdTJCC6KUZqxNKKej3g&pf=p&sclient=psy&safe=off&source=hp&aq=f&aqi=&aql=&oq=draggable+jquery+reset&pbx=1&fp=fca90e9507624f80 http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3... for me, the 10th result is: www.questionhub.com/StackOverflow/3910933 - Cached I think it is safe to say they are scraping, and, while there are some SO answers, that particular answer does correctly interpret what my intention was. In fact, the SO page scraped shows up as the 12th entry.

jmikel 16y ago

Anyone else notice that even Matt's site ranks below other sites who have copied the content after this change? Here's a search for the 3rd paragraph of content in the original blog post: http://goo.gl/vVs8A http://goo.gl/vVs8A Some sites, including this one: http://www.boonebank.com/brc/SBR_template.cfm?Document=headlines.cfm&article=1106 http://www.boonebank.com/brc/SBR_template.cfm?Document=headl... ... which links back to the original post (is that not supposed to acknowladge the original source of the article?) and quotes Matt ranks higher in my SERP results

badwetter 16y ago

Bravo!

Matt_Cutts 16y ago
I'll be curious to collect programming-related queries where we're not returning Stack Overflow or some other site that we should. Computer science and programming queries are easy for engineers to assess and say "Ah, here's something we need to do better on."
- dminor 16y ago
  What's the best way to get these to you (after this article has dropped off the front page)?
  - Matt_Cutts 16y ago
    I'll still circle back to this page for quite a while, or you can tweet them to me. If it's long feedback, you can blog it and tweet a link to the blog post.
    - Natsu 16y ago
      As the guy in charge of fighting webspam, maybe this will help you find or improve upon some algorithmic means of detecting when lots of unrelated wikis start to get exploited to feed up advertisements. http://www.bay12forums.com/smf/index.php?topic=73045.0 http://www.bay12forums.com/smf/index.php?topic=73045.0 Apparently, for a short time (until the admins fixed it), a wiki for a game I play was hijacked to spam Google. Actually, said wiki (magmawiki.com) is under attack right now from spam user accounts. I don't run the place, but I thought I'd pass the information along because it might give you something to examine and because I have to imagine that anything that helps Google identify spam-hijacked wikis would be helpful to everyone.
    - cryptoz 16y ago
      This comment made me chuckle. I agree all this should be discussed out in the open, but blogging and then tweeting links to the blog post....what happened to email!? :)

flawawa2 16y ago

"We" is Google, could someone add that to the title? I was really confused what kind of algorithm would scan submitted pages at HN and rank based on that... On topic: Thanks! I would have thought Google already did that. Don't you also weigh in the age of the content? So if you have seen something earlier somewhere else it would be the "master" and rank higher?

alexsherrick 16y ago

thanks matt this is awesome!

mebigfatguy 16y ago
now if i could only get java 1.6 javadoc to show up, instead of java 1.4.2 javadoc, when i type in any arbitrary java class name.

bhavin 16y ago

I would like to point out one interesting thing I noticed today. I was looking for "gcc optimization flags for xeon". Following query is from google.com and contains no efreedom on the front page. http://www.google.com/search?hl=en&q=gcc+optimization+flags+for+xeon&aq=f&aqi=g-v1&aql=t&oq= http://www.google.com/search?hl=en&q=gcc+optimization+fl... Now, the same query from google.ie (ireland site) contains 2 efreedom on the top page! http://www.google.ie/search?hl=en&q=gcc+optimization+flags+for+xeon&btnG=Search&aq=f&aqi=&aql=&oq= http://www.google.ie/search?hl=en&q=gcc+optimization+fla... Why this strange search behavior to a query which has no relevance to user's location? P.S. I was logged on to my google account while searching, not sure if that has any effect whatsoever.

bhavin 16y ago
UPDATE: I am even more surprised to see that when I logged out of my account and searched again on google.ie, one of the efreedom disappeared!! So, does this mean that Google search now takes into account my past search result clicks (or rather mis-clicks)? Or ranks contents in someway that proves efreedom is somehow more relevant to me?
- Matt_Cutts 16y ago
  If you're using personalized search, we do look at past clicks to change your ranking. So if you clicked on efreedom in the past, that could affect your ranking. You can add "&pws=0" (or use incognito mode in Chrome without logging in) to turn off personalized search and see whether that's the factor for you.
  - andybak 16y ago
    If I'm clicking around to crappy sites to figure out why they rank highly I don't really want them to get even higher in my personal search ranking! Is deleting them from search history enough to correct this effect?
  - bhavin 16y ago
    Adding &pws=0 turned the personalized search off and hence got rid of the efreedom. Also, I cleared my web history and searched again. The results are same now signed in or signed out. Problem solved! :) Thanks!

rgrieselhuber 16y ago

This is great news, thanks very much.

bretthellman 16y ago

Fantastic though I'd rather see google block sites like efreedom all together.

jjclarkson 16y ago
I think that would be a dangerous precedent to set. Whose opinion should decide what sites add value to content that may not be completely original? I'm not saying efreedom is not "being evil", but I could foresee someone somewhere using another's original content and making it more accessible. For example someone could easily improve the accessibility of the content from experts-exchange.com.
- benologist 16y ago
  Google already makes that decision about lots of search results, usage on their services etc.

RealGeek 16y ago

Will this change effect sites like filestube.com and freshwap.net? FilesTube ranks for majority of the long tail keywords, even those not related to downloads/torrents/rapidshare. I see filestube's auto-generated search listing pages ranking on Google all the time. Pages like: http://www.filestube.com/m/matt+cutts http://www.filestube.com/m/matt+cutts http://www.filestube.com/g/google+scraper http://www.filestube.com/g/google+scraper Same goes for freshwap: http://www.freshwap.net/387/dl/Google+Matt+Cutts http://www.freshwap.net/387/dl/Google+Matt+Cutts These sites will give out an auto-generated page for every keyword you enter into it. Apparently, Google loves to index them... there are 126 million pages of files tube indexed in Google. I thought indexing search listing pages of other search engine was against Google's policies.

xenophanes 16y ago
There's some really annoying torrent sites like this. I mean, sites that pretend they have search results for whatever torrent you're searching for. Those show up on a google a lot and they're useless.
- moultano 16y ago
  This is another class of problems we're working on. Expect some changes here in the next few months.
  - skinnymuch 16y ago
    Your account page doesn't say anything. I assume you work for Google?
    - moultano 16y ago
      Yep, in fact I wrote the change we're talking about in this thread. :)
      - skinnymuch 16y ago

tristanperry 16y ago

Sounds good, thanks for the update Matt. I must ask though: I wonder whether you've seen the "January 26 2011 Traffic Change - Back to 'Zombie Traffic' " discussion over at Webmaster World? A number of webmasters there (who own websites with fully unique content) are reporting that they've seen lower quality content sites and/or content scrapers rank above their established sites with good content, starting from around the 26th Jan. There's no specific queries/websites talked about there (I might be wrong but I think Webmaster World has some rules preventing specific discussion of websites/queries), although I thought I'd flag it up since some webmasters have noticed adverse affects from a Jan 26th algo change; and it sounds like this might be the cause. Anywhoo, that being said: it's great to see Google continuing to be on the ball and responding to the recent feedback from various blogs and other sources (e.g. here).

Matt_Cutts 16y ago
I love Webmaster World, but one frustrating aspect of webmaster forums is that specific sites and queries are rarely given. It can be very hard to assess what's really going on since you don't know the specific site that's being discussed.

seanalltogether 16y ago

Matt, I just went through my search history because i remembered a very specific instance of seeing this. Here's the query. http://www.google.com/search?q=nstoolbar+bottom+bar&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a http://www.google.com/search?q=nstoolbar+bottom+bar&ie=u... You'll notice that efreedom.com shows on the first page with content taken directly from stackoverflow. While stackoverflow does show in the results, the exact page that efreedom copies does not. Anyway, I'm glad you guys are taking this seriously. For reference here is what I see right now - http://dl.dropbox.com/u/1437645/googlesearchresult.png http://dl.dropbox.com/u/1437645/googlesearchresult.png

Confusion 16y ago
It's strange how these results vary from person to person. When I click that link, I see an eFreedom link (with SO content) at #6 and a different SO link at #7.
Matt_Cutts 16y ago
It looks like we've got SO above efreedom for that query, but it's always nice to find a url that we didn't have that we'd like to be indexed. That lets us check whether we can improve our crawling/indexing. Thanks for the example!
- millerc 16y ago
  If you tell me that sites who chronically scrape content elsewhere get a "pull-down weight" attached to the whole of their content, you've made my day.
  - bergie 16y ago
    That wouldn't be nice. There is a big difference between copying content from Stack Overflow, to running a Planet news aggregator on a community site for your open source project. The first is an ad scam, the second is valuable service to the members of the community.
- chr15 16y ago

jacquesm 16y ago

Now that's good news. Thanks!

runjake 16y ago

FYI: Don't flag this because there's no link or "citation". Matt Cutts is the web spam guy at Google.

sjs382 16y ago
Here's the accompanying link: http://www.mattcutts.com/blog/algorithm-change-launched/ http://www.mattcutts.com/blog/algorithm-change-launched/
- Matt_Cutts 16y ago
  Yup, I actually posted over here at HN a little bit before I did a post on my personal blog.
  - jjcm 16y ago
    I'd like to mention that this is really an impressive way of working with the community Matt. Companies say all the time that they value their customer's opinions, but rarely do you see a grievance that's posted on a social news site being a.) initially responded to by the guy who's responsible for it, and b.) amended by that person and his team with a request for review. It's heightened my faith that maybe Google can actually keep that small-company-feel no matter how large they get. I really hope you guys continue in this fashion, and thanks for the search fix.
    - yuhong 16y ago
      Yea, I have recently posted a comment starting with "Any Google employees here" knowing that there is more than one employee that posts here.
    - Matt_Cutts 16y ago
      HN has been a really high signal/noise site in discussing these issues, so it only seemed fair to give folks a heads-up here and see what other issues people were seeing. But thank you. :)

miah_ 16y ago

This is awesome! Now if only I could completely remove certain sites from my search results ala the Google Wiki stuff. I'd love to drop swik, and expertsexchange and a few other annoying sites. Its possible this algorithm change will make these sites less annoying to me though.

SkyMarshal 16y ago
Second that. WTB ability to customize my search results by blacklisting certain sites. Big change I know, but would be a hugely useful service Google could provide.
- PaulJDrago 16y ago
  why not just use the - function? Create something like: http://www.google.com/search?q=google+-google.com&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a http://www.google.com/search?q=google+-google.com&ie=utf... I suspect you could hack together a quick greasemonkey script to automatically inject your negative site list when the URL is generated
  - miah_ 16y ago
    Because Chrome still doesn't sync your "search engine" settings. So maintaining a massive customized search engine across the multiple computers I use would be a pain. If this was a native feature it would work so long as I logged in to Google. Of course, if they fix chrome I'll be happy too.
  - Devilboy 16y ago
    This only works for a small number of blacklisted domains.
- mythobit 16y ago
  I created a site that uses Google CSE to do just this. The link is: http://blacklist-search.appspot.com/ http://blacklist-search.appspot.com/
- 16y ago

WillyF 16y ago

Matt, this is great news. How about sites that rank well with no content, just navigation? Here's an example: http://www.collegegrad.com/entryleveljob/entrylevelaccountingjobs.shtml http://www.collegegrad.com/entryleveljob/entrylevelaccountin... It's generally a high quality site, but that page has absolutely no relevance to the query except for a title tag and some internal anchor text. The search terms aren't even on the page. If I remember correctly, it used to rank #1 for "accounting entry level jobs," and now it's down to #8. My question is why is it even ranking at all? It's not even low quality content. It's no content.

deleted 16y ago
[deleted]
tivaceous 16y ago
I notice that you run a competing site to collegegrad.com - namely onedayonejob.com. Obviously you are likely to have paid attention to some of the things your competitor has been doing. But are you sure you're not just using this as an opportunity to stick it to them? If so, that would strike me as a distasteful use of this forum, especially since Matt has been very gracious to give this opportunity to the HN community.
- wizard_2 16y ago
  Even if they were direct competitors with the exact same market and product (Which they don't appear to be), that could mean that he's got a much better idea of how his competitors may be doing spammy things then we do. Since these are algorithmic changes not site specific changes any resulting fix would get applied to his own site as well. The motives here don't offend me.
- WillyF 16y ago
  I completely understand why you'd question my motives. I thought really hard about whether I should post this example or not. I tried to find an alternative example that would demonstrate the same problem, but this is the only one that I could come up with. I'm sure if I spent significantly more time looking I could have found something similar, but this example is very apparent to me because it's in SERPs that I watch closely. I'm not trying to "out" this site. I don't think they've done anything wrong or manipulative—they just have some pages with no content that are ranking well. I think that this type of issue should be on Matt's radar. Most of the site is of extremely high quality, and it deserves the high rankings that it gets. It's a page that is solely navigational structure, yet it ranks very well for a very specific keyword because of title tags and anchor text. We've already seen that Google has a problem (that they're working towards fixing) with low content from low and medium quality sites. What are they doing about low quality content (or non-content) from high quality sites?

spidaman 16y ago

Nice to hear about this change Matt. "Fuckin' efreedom..." is an oft heard missive around our office, godspeed in ranking them and their ilk down.

Aaronontheweb 16y ago

Glad to see that Matt went out of his way to help us geeks and the StackOverFlow community! In the future is there going to be some way for webmasters to do something like rel="canonical" across domains so if I want to syndicate a piece of content across two properties I own I can indicate which one is the original source? My understanding is that rel="canonical" is only meant to be used between pages on the same root domain today but I could be mistaken.

sjs382 16y ago
rel="canonical does work across domains.
- Aaronontheweb 16y ago
  Good to know, thanks! I wasn't sure if that was the case or not. Is Google the only engine that supports it currently or is it standardized?
- gregable 16y ago
  http://googlewebmastercentral.blogspot.com/2009/12/handling-legitimate-cross-domain.html http://googlewebmastercentral.blogspot.com/2009/12/handling-... was the official announcement.
moultano 16y ago
Syndicating content across your own domains is a common practice and will not be effected by this.

brown9-2 16y ago

Is this change restricted to programming-related queries only? I noticed today that a search for "mubarrek london" returns a page of results where every result on the first page, besides the top one, is spam from www.88searchengines.com, www.30searchengines.com, www.70searchengines.com, etc. I know this might not be related to topic of scraper sites directly, but not sure how else one can easily report these types of things.

petervandijck 16y ago
Google doesn't link to search engines in its results, that's their policy, so I would think those sites will be removed from their index at some point.
treeface 16y ago
I can't seem to replicate this: http://i.imgur.com/kDwPd.png http://i.imgur.com/kDwPd.png Are you sure there isn't something else going on?
- brown9-2 16y ago
  Perhaps it's related to location, it's reproducible with incognito search (to make sure no personalized settings are being applied): http://i.imgur.com/Yd7FN.png http://i.imgur.com/Yd7FN.png
- cd34 16y ago
  http://cd34.com/mubarek+london.png http://cd34.com/mubarek+london.png I get similar results as he does, with personalization, and, in Incognito mode.
- tristanperry 16y ago
  I do see these problematic results: http://www.tristanperry.com/pics/SpammyResult.jpg http://www.tristanperry.com/pics/SpammyResult.jpg Searching from the UK, if that changes anything.
boundlessdreamz 16y ago
Tried the query and was able to replicate this https://skitch.com/techsutra/rm3gb/mubarrek-london-google-search https://skitch.com/techsutra/rm3gb/mubarrek-london-google-se...

giberson 16y ago

I imagine theres not much you can be specific about when talking about google's algorithm, but can you at least disclose if the identification of "original" content providers is "determined" (by some automatic process) or is it "specified" (manual intervention)? Stack overflow is an obvious benefactor of this new change, I'm just wondering if smaller content providers might benefit as well?

moultano 16y ago
There's nothing manual about this. Everybody who authors their own content should be helped. :)
- Enlightenment 16y ago
  Please take care of the wikipedia scrapping trolls? http://meta.wikimedia.org/wiki/Live_mirrors http://meta.wikimedia.org/wiki/Live_mirrors http://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks http://en.wikipedia.org/wiki/Wikipedia:Mirrors_and_forks http://openfacts.berlios.de/index-en.phtml?title=Copies_of_Wikipedia_content http://openfacts.berlios.de/index-en.phtml?title=Copies_of_W... http://en.wikipedia.org/wiki/Category:Websites_which_use_Wikipedia http://en.wikipedia.org/wiki/Category:Websites_which_use_Wik...

jjclarkson 16y ago

I ran across this list just now on sites that may be using SO content: http://meta.stackoverflow.com/questions/24611/is-it-legal-to-copy-stack-overflow-questions-and-answers/48962#48962 http://meta.stackoverflow.com/questions/24611/is-it-legal-to...

eurohacker 16y ago

may be here is a good place to ask - may be someone can answer - what does the term "original content" actually mean , original in what sense ? if you have a blog about wines, then what do you need to write - in order to be "original" blog about wines - does it mean mostly: 1) express original opinions about wines, 2) original structure of sentence and original wording - but the same opinions that 10 other sites write about, 3) original brands of wine you talk about - original names mentioned, 4) original content in the sense that you have not copied the text 100% from someone else - or what 5) combination of the factors mentioned above 6) something else that makes your content original ..

socialmediaking 16y ago
In order to be considered "original" from the search engine perspective, it needs to be around 30% new content. There are websites like http://www.copyscape.com/ http://www.copyscape.com/ that can tell you how original something appears. Internet marketers use software to "spin" text and make it appear unique. Sometimes the text is annoying for humans to read, but search engines eat it up. You can take one article and turn it into 50 with the click of a couple of buttons.
lallysingh 16y ago
I think this is a much simpler content, as in not being a literal copy of another's text. To be an "original" wine blog, write your own blog entries. Don't copy someone else's.

deleted 16y ago

[deleted]

hsmyers 16y ago

While I applaud such an improvement, I still despair over Google's inability to handle context. For instance the search phrase 'tex decorative rules' is totally miss managed. First it overrides the search and changes tex to text. Humorously enough, when you counter override back to tex, the search is even worse. We won't even speak of what happens when you change tex to latex--- porno doth ever rise to the top I guess. While we may be in a new millennium, some things remain far behind. And yes I know that things are being worked on from the those who were academic and are now working at Google, but still...

Matt_Cutts 16y ago
tex is a hard one. And [latex decorative rules] is even trickier--most people wouldn't expect technical documentation if they just saw that as a random query. That's more of an issue to pass on to the synonyms team and not in the scope of this change, but I'll pass this one on to the right folks
- runningdogx 16y ago
  Would it be feasible to introduce a search customization setting: "lower the estimated probability that I mistyped"? I assume google uses something like two cut-offs: A < B, where if P(mistype)>A, it suggests but doesn't search for the correction, while if P(mistype)> B, it displays auto-corrected results. Simply an user option to nudge P(mistype) lower would probably suffice, or an option to increase B. A major usability problem for me with auto-corrected results is the two lines with links: "Showing results for <link>" and "Search instead for <link>". Interpreting those two lines takes me out of the flow of analyzing search results. I haven't yet been conditioned to click on the second link. I never want to click on the first link, because google is already showing me those results. I frequently search for unusual acronyms, terms, variables, made-up words, and gibberish that shows up in logs, among other things that google likes to "correct" for my benefit. I realize those use cases are not typical, but current auto-correction behavior can be extremely aggravating in those cases. Auto-suggest a correction all you want, but auto-correcting and suggesting the original is going too far I think, unless there are zero hits, or some equivalently strong algorithmic determination is made that I couldn't possibly want the query the way I typed it.

MelissaLB 16y ago

will there be any penalization for sites that use wikipedia content. Specifically a site that would NOT show up in the same search results as a wikipedia page but a commerce site that uses some unaltered wiki content in the descriptors of the products? I've seen many sites use this method of adding text to an image heavy site and have watched one particular site that uses this method of adding text, drop dramatically out of the search results since October 28th and again around Dec 28th. Would you advise to discontinue the use of wikipedia content even though the targeted keywords differ from those of the actual wikipedia page?

MelissaLB 16y ago

Alex3917 16y ago

Not exactly a scraper site, but if you do a search for "learn to hack" the top result is just a list of SEO keywords: http://www.learn-to-hack.com/ http://www.learn-to-hack.com/ Several of the other results are rather dubious as well. The reason I bring it up is because the Squidoo lens that comes up is something I made, and while certainly not perfect it's still a much better than many of the SEO spam sites and fake eBooks that rank above it. (And plus the ad revenue is going to charity rather than some shady organized crime ring.) Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works.

moultano 16y ago
>Anyway sorry if it's a faux pas to complain about my own stuff, but I feel like it's a legitimate problem with the way Google works. Definitely not a faux pas. Thanks for the example. My biggest annoyance in threads like these is people who write essays about their site losing traffic but then aren't willing to provide an url for people to check out.
- Enlightenment 16y ago
  The guilty don't want google to know what they are doing <grin>. Keep up the good work squelching the trolls!

deleted 16y ago

[deleted]

angelbit 16y ago

Hi Matt. Why Google don't test the results after changes on algorithm? A simple program that store and compare n SERP results before and after the changes.

Matt_Cutts 16y ago
We do.
- fzk390 16y ago
  I still see sites with low quality duplicate content but highly stuffed keywords. Here's one such example http://www.google.com/search?hl=en&q=internet+phone+service http://www.google.com/search?hl=en&q=internet+phone+serv... Check the listing for www.internetphoneguide.org Also for http://www.google.com/search?hl=en&q=voip+phone+service http://www.google.com/search?hl=en&q=voip+phone+service see the listing for www.zimbio.com which is not an authority on the topic. Last but not the least http://www.google.com/search?hl=en&q=home+phone+service http://www.google.com/search?hl=en&q=home+phone+service The site www.freelifelinephone.com jumped to Page 1 over night with the latest update.

hananc 16y ago

Please don't take my word for it. Hit http://duckduckgo.com http://duckduckgo.com with your programming queries. No - I am not affiliated and am not getting paid to write this.

iPadDeveloper 16y ago
Then why the advertisement? This is about Google's search result algorithms.
- WilliamC 16y ago
  Because he couldn't find a better place to spam with traffic.

Nickwiz 16y ago

Hey matt, thanks for the update I wanted to know how would it work in this case: http://www.google.com/#q=major+online+dating+sites+koopa&hl=en&client=firefox-a&sa=N&rls=org.mozilla:en-US:official&fp=1 http://www.google.com/#q=major+online+dating+sites+koopa&#38... We submitted an article to ezinearticles from our blog.koopa.com. Just wondering how our blog is not remotely listed but ezine which we posted are article through our blog has? How does this work when our authors submit to article directories? On that same note I notice a bunch of other sites just below which have copies ezines article to the tee and are ranking higher then the our original blog that posted it. Is this related to the current algorithm change or just that our blog may not be indexed yet? Thanks for the update once again. I love the fact that you and the google team are constantly updating and changing your algo to give value to rightful content owners.

moultano 16y ago
What is the url of your blog?
- Nickwiz2 16y ago
  http://blog.koopa.com http://blog.koopa.com

dcdan 16y ago

At what point does Google cross over from "ranked by algorithm" to "ranked by algorithm selection as editorialized by Googlers and bloggers?" Publicly discussing algorithms changes like this seems like a potential PR problem.

Matt_Cutts 16y ago
The whole history of Google is trying to find algorithms that encode our philosophy and mental model of what we think users want. We've been discussing algorithmic changes with people online since 2001, when GoogleGuy would show up on webmasterworld.com to dispel misconceptions.
- dcdan 16y ago
  This algorithm change could come across as more editorial and less empirical. It's publicized as highly targeted ("slightly over 2% of queries"), responding to small tech discussion (Atwood, SO, HN, quality launch meeting), and seemingly rolled out in a short period of time. The media could easily boil this down to "a small number people felt sites they liked were under-ranked, so Google moved them up a week later." I mention this because Google often talks publicly about being entirely algorithmic, and elements of this narrative feel human.
  - nostrademons 16y ago
    2% of queries is huge. If you average that across a population (which of course isn't how it's actually distributed...), a searcher could expect to run across such a query every few days.

nikothefinn 16y ago

Since Matt is responding here, I figured this is worth a shot, no harm in asking. Matt, would love a response from you if you get a chance, since the Webmaster Tools appeals process gives no insight whatsoever to our situation. Following on from one of the comments here, namely the idea that "value is in the eye of the beholder", I'd like to raise our own plight. I run a number of aggregator sites - the largest and oldest of them being celebrifi.com, which was a PageRank 5 until Google de-indexed us in December (along with some of our other sites, but interestingly not all). A little background - the purpose of the sites is to aggregate, organize, rank and add context to what's happening in the news, with each site focusing a specific vertical. Think Techmeme, but with more context. I'll be the first to admit, there is no original content, but I strongly believe that we "add value" by figuring out what exactly is going on in any given story or blog post. We add value to publishers, by always linking to the original source (indeed, many publishers directly request that we add their feeds to the sources we track), we respect copyright by only displaying a short snippet of the original text and only displaying thumbnail images and we add value to users by giving them easy access to a lot more content on the same topic/story, all in the same place. Google's Quality Guidelines clearly state that duplicate content is penalized, and that is totally fine with us, but is it right to totally de-index a site for duplicate content? I wouldn't even want to rank above the original source for any given piece of content, as I respect the hard work that writers and publishers put into creating quality content, but aggregators who add value have a role to play in the content ecosystem. Digg, for example, uses the "wisdom of the crowd" to aggregate and rank content - hence adds value. Topix takes a local approach to aggregating content, and uses comments to rank content - hence adds value. We take a verticalized approach to aggregating and ranking content, and hence I believe that we add value. As mentioned above, we got de-indexed in December, and despite going through the appeals process, fixing a few things on our end to do with sitemaps, and clearing out some older "low quality" sources that we were tracking, we received no clarity into what our crime was. Matt, I'd like to raise this issue with you - both as it relates to us, but also as a general industry question - are all aggregators going to be de-indexed? And if not, which aggregators are and which aren't? What is the criteria, and who decides? If its algorithmic, then I am very curious to know what on our sites triggered the de-indexing? And even more curious to know why some of our sites got de-indexed, and some didn't. I have great respect for Google's efforts to clean up spam and low-quality content - and would always expect to see original content ranking higher than aggregated content. But to completely de-index an established aggregator site and strip it of its PageRank seems very draconian. I would love to hear your/Google's position, and look forward to some more clarity on both our situation, and the future of news/content aggregators. Respectfully yours, Niko

akie 16y ago

Yes - I've been doing a lot of YII related searches the past week, and I've noticed that a lot of times a site called 'devcomments.com' pops up somewhere in the first 5 results. Usually with a bogus page that does contain the keywords you were looking for, but not the actual discussion/forum thread. It appears they have copied their content directly from the official site, yiiframework.com. Example: http://www.google.com/search?q=Any+yii+way+to+get+the+previous+URL%3F http://www.google.com/search?q=Any+yii+way+to+get+the+previo... In that particular instance it is result number 3 (and the original is number 1), but on more than one occasion it was the top result and it's never what you are looking for.

samd 16y ago

I don't suppose you'll tell us how you know whether a site's content is original or not. What if you have a blog with lots of quotes from other sites; will that hurt your rankings because Google sees "unoriginal" content? Is there some ratio of original to unoriginal content that must be met to keep from being flagged as a scraper?

kqueue 16y ago

Here's a query that shows efreedom above SO. I did site:efreedom.com and picked few titles until I found one. query: Mailengine with .NET API The right page that should show up is http://stackoverflow.com/questions/1720900/mailengine-with-net-api http://stackoverflow.com/questions/1720900/mailengine-with-n... If you google the link below you'll see that the page is indexed. http://stackoverflow.com/questions/1720900/mailengine-with-net-api http://stackoverflow.com/questions/1720900/mailengine-with-n... However, the page doesn't show up when you use the query I mentioned.

seles 16y ago

This is good news, but I feel like it is only treating a symptom not the actual disease. If the algorithm properly detected site relevance, importance and viewer satisfaction, those copycat sites should never have ranked higher in the first place. In a way this is admitting that it is impossible to stop the gaming of search engine optimization, and that the only way to deal with it is to "win" in some special cases. That being said I provide no real solution, this is a huge problem with millions behind each side. Although this is good news it is also gloom news.

LiveTheDream 16y ago
How would an algorithm detect viewer satisfaction?
- seles 16y ago
  How long user stays on page (hint would be how soon they make next search/click), whether or not they continue looking for something under a similar search query, having actual buttons users can click to rate, etc.
  - moultano 16y ago
    Spam gets upvoted to the top of reddit on a regular basis. Most people are pretty apathetic to legitimacy most of the time. All we ever have is proxies for quality, the hope is that we have enough of them to cover the space.

kunjaan 16y ago

I googled for my own name "Kunjan Kshetri" and I still see sites like 1. www.peekyou.com/_kshetri 2. www.yasni.com/kunjan+kshetri/check+people 3. mobile.aa.com/mt/net35.ccs.neu.edu/home/kunjan/?un_jtt in the front page. These sites reused my homepage or just have my name listed. They too add no original content. Please remove these sites. Please note that searching in Bing produces more original content results.

brianpan 16y ago
Google doesn't just remove sites from their index. They will always be there. But if you think a different site should rank higher than these spammy ones, you should mention it and maybe someone can look at it. (I don't work for Google.)
- kunjaan 16y ago
  I meant remove them at least from my front page.
  - porges 16y ago
    For a while there was a Google feature that let you do that, along with up/downvoting results. It got removed though, and I've never read why :/

avstraliitski 16y ago

Very happy to see the end of copied Wikipedia sites.

klbarry 16y ago

Hi Matt, I'm sure your busy and don't want to answer all SEO related questions anyway. I wanted to ask, though, what Google things about the importance of exact anchor text in rankings as remarked upon in http://www.seomoz.org/blog/how-organized-crime-is-taking-control-of-googles-search-results http://www.seomoz.org/blog/how-organized-crime-is-taking-con...?

EGreg 16y ago

I got 99 comments but of which this ain't one :)

endergen 16y ago

Matt would you comment on the thought process of Google on curation. There are sites that are generally strongly disliked. Examples: Demand Media in general. Or say w3schoolS.com for JavaScript/CSS/HTML. I can understand that there would likely be a deluge of legal action against Google if this was done in a heavy handed way. Or is it a principled thing where everyone should be treated equal? If so isn't that a lot of algorithm ideology. In practice hard rules without human judgment leads to ridiculous edge cases or bypassing the intent of the rules. I always assumed you curated in a way via having teams that specialized in creating topic specific search techniques which then all get combined together via topic detection or some other meta algorithm. This would be a good balance between manual curation and having it be a scalable approach.

bkaid 16y ago

While you are at it, can you go ahead and de-index this site: http://www.google.com/search?q=site:livestrong.com http://www.google.com/search?q=site:livestrong.com

23david 16y ago

Will this new change impact the issue where scrapers that take videos and video descriptions from youtube and turn them into 'blog posts' show up higher than the youtube page that contains the actual original content? I worked for nearly a year producing a few hundred videos only to find that spam sites (usually running adsense ads) were showing up ahead of the video on youtube.com and my own site. Our site's domain name was even plastered all over the description and it still didn't matter. The spam sites still showed up way ahead of us in the search results.

tmsh 16y ago

One caveat, though I imagine this has been thought of before, is that mobile versions of sites often have the same content as full-browser versions of sites. So ideally, perhaps m.google.com would be able to sort through this and not penalize the duplicated-nature of the mobile version.... Anyway, something to think about if you haven't already.

jswinghammer 16y ago

Ok here is the search that first brought this issue to my attention: mysql spatial index example lft rgt I see the following order: 1: http://planet.mysql.com/entry/?id=23512 http://planet.mysql.com/entry/?id=23512 2: http://efreedom.com/Question/1-1743894/Mysql-Optimizing-Finding-Super-Node-Nested-Set-Tree http://efreedom.com/Question/1-1743894/Mysql-Optimizing-Find... 3: http://explainextended.com/2009/09/29/adjacency-list-vs-nested-sets-mysql/ http://explainextended.com/2009/09/29/adjacency-list-vs-nest... 4: http://stackoverflow.com/questions/1743894/mysql-optimizing-finding-super-node-in-nested-set-tree http://stackoverflow.com/questions/1743894/mysql-optimizing-...

deleted 16y ago

[deleted]

gbesen 16y ago

Hi Matt, I would like to ask if vertical search engines like vast.com is considered as a scraper site? Real estate search, used car search and other vertical search concepts do need traffic from google. I for one find it valuable to visit a vertical search engine that saves me from visiting site after site to find the car / house I want. Does this mean the end of vertical search engines? (I'm not affiliated with vast.com in any way) I'm sure many people would like to know about this, thank you.

Galaxis 16y ago

My current favourite spammy search (Windows process names) seems slightly improved, but still produces mostly unusable results. Try something like: http://www.google.com/search?q=hidfind.exe http://www.google.com/search?q=hidfind.exe Another similar search area (Windows drivers for various hardware) has improved a lot judging from the few examples I just tried - actual hardware manufacturers are now amongst the top results for a change. So that's quite a step forward...

quag 16y ago
Repeating a quick search I was doing a week or so ago to find drivers, I'm not seeing an improvement. Asus actually has a great driver site, that is way better and less hassle than the spam sites, but I've yet to see it show up in search results. I only found it because one of the spam sites was using asus's servers to serve up the drivers. http://www.google.co.nz/search?sourceid=chrome&ie=UTF-8&q=sis+asus+motherboard+drivers http://www.google.co.nz/search?sourceid=chrome&ie=UTF-8&...

kqueue 16y ago

The fact that efreedom results are showing up in the results is irritating by itself. We all know efreedom is spam, and so does Google. Now I am always on alert when clicking on a link in the results to avoid the spam pages like efreedom, expertexchange and whatnot. Why not just remove them from search results? They are causing enough bad publicity to Google.

forgotAgain 16y ago

When I search for "rabbitmq exchange declare" (no quotes) I noticed the following. Mailing list entries from http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-... show up from old.nabble.com before the original source. From a quick look there is significant improvement.

JWilder 16y ago

matt, kudos! this is great news. a question that I always pondered, what is googles approach on more clever forms of rehashed content that involve photoshopping / cropping an original image, is this something that google looks at? Example source photo: (http://shoes.n-sb.org/img/thumbs/472153d5fc674fa1c685f3c7814ecee9.jpg http://shoes.n-sb.org/img/thumbs/472153d5fc674fa1c685f3c7814...) rehashed image: (http://images.sneakernews.com/wp-content/uploads/2011/01/nike-sb-blazer-low-end-theory.jpg http://images.sneakernews.com/wp-content/uploads/2011/01/nik...)

cperciva 16y ago

searchers are more likely to see the sites that wrote the original content This is great news, but I have to wonder: How do you figure out which site wrote the original content? I'm wondering based on what happened to Tarsnap last week, where hackzq8search.appspot.com outranked tarsnap.com on a search for "tarsnap".

MiaBellezza 16y ago

http://www.homebasedesign.com/unusual-pergola-designs-deck-and-unusual-pergola-pictures-by-miabellezza/ http://www.homebasedesign.com/unusual-pergola-designs-deck-a... this site has copied numerous sites on the net and though Google adsense was taken away, it still ranks well. No page is their own. All are copies. The link above is referencing a search for unusual pergola designs, of which my lens is first in search. It has copied my whimsical garden art, etc. These are mine and the rest are copies of other people's sites. Though they never seem to update their garbled text and offend with blown up pictures, they still continue to rank well in searches.... Simply Amazing and this situation has been ongoing for a year. Yes I asked them several times to take out the pages to no avail. The whole site should be taken down. It's a joke. here is their copy of my Whimsical Garden Art: http://www.homebasedesign.com/whimsical-garden-art-ideas-for-your-whimsical-garden-by-miabellezza/ http://www.homebasedesign.com/whimsical-garden-art-ideas-for... my lens that they copied: http://www.squidoo.com/whimsical-gardens-art http://www.squidoo.com/whimsical-gardens-art There are other sites they created and did the very same thing: homedesigndecorating.com hometrendesign.com onhomedesign.com hahoy.com no original content, all copied. MiaBellezza