9 ms·
Google rolls out algorithm change in the US
- izendejas 16y agoBrilliant use of crowd-sourcing to gather a test set via the chrome plug-in. Makes absolute sense, otherwise you can overfit. Great turn-around time also.
- deleted 16y ago[deleted]
- abraham 16y agoFrom the article: > It’s worth noting that this update does not rely on the feedback we’ve received from the Personal Blocklist Chrome extension
- joshzayin 16y agoThe implementation of the algorithm didn't rely on it, but they did test it using that data. >However, we did compare the Blocklist data we gathered with the sites identified by our algorithm, and we were very pleased that the preferences our users expressed by using the extension are well represented. If you take the top several dozen or so most-blocked domains from the Chrome extension, then this algorithmic change addresses 84% of them, which is strong independent confirmation of the user benefits.
- deleted 16y ago[deleted]
- izendejas 16y agoedited: to respond to abraham's post directly There's a difference between training with that data and testing against it, which is precisely what I wanted to highlight here since many expected Google to use such data for training. You can't do that since you can overbias/over-fit given what little daat we, plug-in users, are able to produce versus the amounts they have access to. 2nd edit: it should also be noted that that 84% figure they mentioned related to the extension data is a measure of recall. Assuming Google wants to do a better job with precision in these cases (don't want to get many false positives), that recall is still fairly good in mind since I'm sure there's some noise in that data.
- darksaga 16y agoFor me, it's too little too late. I know a lot of people who have already moved to different search engines. Google lost a ton of money in the process. They didn't help themselves any when it was revealed JC Penny had successfully gamed Google for almost an entire year - and Google had no idea until the NY Times broke the story.
- trezor 16y agoWhile I don't agree with the opinion expressed or the tone it is expressed in, I do think the JC Penny story sound interesting. Personally not having heard of it before, I think this comment (accidentally?) contributes somewhat to the discussion. Has HN really become this trigger-happy on the downvotes?
- weego 16y agoYou may well know a lot, but I bet 100% of them are serious tech interest people which accounts for a tiny tiny % of the internet. You could argue that tech people drive the trends that spread out to the masses but in this case I think it's just bashing the big guy cos he's an easy target. The idea that most people really care about the purity of their search results, let alone would understand what all this means even if the BBC or NYT ran a proper article is just unrealistic.
- iskander 16y agoWhat other search engines have your friends moved to?
- ladon86 16y agoIs the change live yet? I remember an example provided here was "nstoolbar bottom bar". Let's take a look: http://www.google.com/search?q=nstoolbar+bottom+bar http://www.google.com/search?q=nstoolbar+bottom+bar Granted, it was probably linked from HN somwhere which will have made matters worse.
- rksprst 16y agoYes, the change is live (rolling out slowly through the day). A ton of large sites have already been affected. It's a pretty big change.
- A1kmm 16y ago> sites that copy others’ content It would be interesting to know how Google determines, in an automated perspective, which site is the original and which is the copy - especially when copying could go both ways. For example, I could write an article, license it under the GFDL, and someone could copy it to Wikipedia. I might then copy the Wikipedia improvements back to my site. Technically, I had the content first - so would Wikipedia be penalised? If there is a bias against smaller sites, this might make smaller sites be reluctant to license their content under licenses that let bigger sites copy them.
- Devilboy 16y agoThese days they crawl sites often enough to know where the info appeared first.
- alextgordon 16y agoMy 4am thoughts on the matter... If a site has N copies associated with it, then you could compare that to the average number of copies associated with sites of that pagerank. If the difference between N and the average is high, then that's a likely spam site. Let's call this difference σ. This gives us a problem though, because while σ is a good indicator of spamminess, it's not foolproof. A low pagerank site could have been copied from lots of times, which would unfairly earn it a high σ. What we can do is calculate a σ', by inversely weighing each copy with the σ of the site on the other end. A copy of a site with a low sigma will increase σ', and a copy of a site with a high sigma will have less of an effect on σ'. So while our high σ may have initially suggested the site to be spammy, all its copies are from high σ sites, which are counted less towards σ', leading to an overall verdict of not spammy. [I wonder what would happen if you iterated this process] As for your example, the smaller site wouldn't be penalized, because N would be low. If it were say, a full wikipedia mirror, then it would be penalized. Wikipedia would not be penalized, since it has an ungodly amount of pagerank. There's also no bias against smaller sites, since σ is calculated relative to pagerank. What have I missed?
- DarkShikari 16y agoThis is a huge problem with Wikia. Wikia has a relatively high pagerank. Sometimes, because of Wikia's often-abusive policies, communities will decide to move their wiki off Wikia and host it separately. But Wikia will refuse to remove the original wiki even if all the contributors want to move it, in order to maximize advertising revenue. This means that it's nearly impossible to get the new wiki to rank highly on Google, even if all links across the internet are changed to point to the new wiki, because Wikia's pagerank is so high that Google deems the new one to be a "copy". This is why Wikia moved all their wikis to subdomains: in order to piggyback on the pagerank of the main site. As a result there are a ton of long-dead wikis on Wikia that still get more search traffic than the active equivalent. Obviously this hurts users, since they get long-outdated information as a result. In short, once a wiki is placed on Wikia, it's basically impossible to ever move it anywhere else because of Google's anti-duplicate biasing. See http://en.wikipedia.org/wiki/Wikia#Controversy http://en.wikipedia.org/wiki/Wikia#Controversy for more info.
- bkaid 16y agoIt'll be as interesting to see how this affects Bing rankings.
- joshhart 16y agoIt should affect them automatically. We know Bing uses Googles search -> page results coming from the Bing toolbar. If a site's ranking is reduced by Google, the same should happen (maybe to a much smaller extent) in Bing since that feature is now smaller.
- trezor 16y agoIt doesn't have to and I believe you are mistaken/oversimplifying. As far as I understood the whole Bing ordeal was that users with the Bing toolbar reported not the links and ranking shown, but what the users chose as the "correct" hit for that search. In that regard this doesn't really have to alter Bing's results in any way at all.
- JacobIrwin 16y agoGoogle works hard (or maybe less) to keep their front page UI ultra clean. I do think users could benefit a lot from a small drop down arrow under the search field that stretches open another field to input the words searchers do not want included in their results. Most of us in the Hacker News community know this can more easily accomplished by including a "-" (minus sign) before such terms; we know how to use "-" and "site:..." to narrow query results. However, I'd say 50% of Google users don't know these tricks and waste time navigating to the advanced search link.
- scottmp10 16y agoThere is an advanced search link next to the Google search bar that provides the functionality you described for non-power users.
- JacobIrwin 16y agoI mentioned that in the last line. I just feel it's not efficient for those who want to leave out certain terms... being that "-" is the most common query alteration (or maybe second after "+".
- bryanh 16y agoNot to nitpick, but... If you take the top several dozen or so most-blocked domains from the Chrome extension, then this algorithmic change addresses 84% of them, which is strong independent confirmation of the user benefits. Well, what if they are mistakenly penalizing Common Jack's blog because it gets re-posted across the interwebs? I bet no one is manually blocking Common Jack's blog, so that factor is lost. Example: let's say Google just randomly penalizes 84% of ALL domains. Chances are that they intersect with, you guessed it, 84% of the Chrome extension's data. Independent confirmation!
- mtkd 16y agoThe problem for Google is that any non-algorithmic approach (using a feedback mechanism) will just be abused by the SEO industry - they'll just crank up mechanical turk to start reporting anyone ranking above their clients.
- bryanh 16y agoI'm not advocating that, I'm just saying that the "independent confirmation" tells us little about the innocent sites that get negatively affected. They are, by definition, excluded.
- Dylan16807 16y agoAnd this somehow affects only 12% of searches?
- lutorm 16y agoTrue. It tests for completeness, not purity.
- invisible 16y agoThat is not what that said at all. It said that of the 36 or so domains that were the MOST blocked, 84% of those sites were lowered in the search results. Which means 4 or so crappy domains escaped being penalized. With that high of a "win" they might affect 1 or 2 domains on the entire Internet that should not be affected but I am sure it will eventually be corrected for those domains. This also means that many more than 36 domains will get punished for poor tactics.
- codefisher 16y agoIt looks like I have got higher up the ranks for some of my keywords, so the change must be good ;)
- zone411 16y agoI know this is not a popular opinion here, but I'm not sure if I like Google reacting to what tech bloggers echo chamber talks about. Are we sure that this what is good for regular users and for the creation of quality content on the Internet? This group is not representative at all. There seems to be enough "normal" spam in the search results that Google should be focusing on first. Just yesterday I searched for "viking dishwasher clog" and the #6 result is a .info page that is nothing but obvious spam. I previously mentioned the issue of Wikipedia poorly regurgitating content from original creators (sometimes with attributing, sometimes without) and outranking that original page, even if it contained approachable, illustrated and in-depth article, and Wikipedia's content was much weaker. This is not something that tech bloggers will notice - they are much more likely to read Wikipedia's programming, science, math, or tech articles, which are of much higher quality than those on other topics, so you won't hear them complaining when they see Wikipedia as #1 everywhere. Another bias I starting seeing lately is the high ranking of new Q&A sites (another focus of tech bloggers), when excellent topical forums with very good content, often exactly with the answers I needed, are ranking poorly. I’m speculating here, but I think Google’s focus on links might hurt its ability to bring up deeply hidden content from forums. These forums won’t get many links from tech bloggers and others looking for the next big thing, but they have very valuable content for many long-tail searches.
- teoruiz 16y agoIs there any reason to change the algorithm, initially, only in the US?
- jbk 16y agoI really hope it will change for the better, but I don't trust Google anymore: Small example: When you search "VLC" on google on find scams websites shipping VLC with a toolbar/crapware/malware, buying or not adwords... Google plainly refuses to remove those websites, because they gives Google some money. So they make money from scammers, know it clearly and do not want to do anything... And as we are a very small team of volunteers, it is impossible to attack those websites... NB: this happens with other software too.
- gimpf 16y agoStrangely enough, your primary example always directed me to videolan in the past, and continues to do so. I very much prefer DDG these days, but those statements are just not true. And I do not believe that Google would purposefully provide _much_ _worse_ results for US customers, relative to EU ones.
- CWuestefeld 16y agoGoogle plainly refuses to remove those websites, because they gives Google some money. You may be right about what's happening, but I don't think it's fair for you to assert as fact what you believe their motivations to be. Matt Cutts and others have categorically denied these claims, and provided good reasons.
- jbk 16y agoThen, I am all ears to why Google refuses to take down, when asked, scam pages of open source products with toolbars and other adware/malware.
- nikcub 16y agoWhen Pagerank was created, it worked really well to find the most relevant sites. The old search engines such as AltaVista and Excite were drowning in spam since the SEO experts had figured them out. Does anybody else feel that Google has been figured out now, and that they are now just applying patches to a system that is fundamentally broken (in the same way Excite and AV did)? There might be an opportunity here for a new search engine - one that creates a new core method of ranking, as PageRank was, to filter out duplicate content, content farms, malware sites, etc. "Relevance = links" today looks as tired as "relevance = keyword density" did 10 years ago, but I have no idea what the new "relevance = " is
- arkitaip 16y agoGoogle is the best company to figuring out this "new" search engine. It's vastly cheaper and easier to continue tweaking their current system than it would be for someone else to code a search engine from scratch. IMHO there exists no silver bullet approach to search, instead it's a on-going quest/war consisting of infinite trials and errors. Also, I am not sure that the spam problem is as bad as people claim it to be. I have looked at virtually all of the lists of content farms that have been published since Google release their Personal Blocklist for Google Search and it seems to be that (a) it's all very subjective and people rarely agree that a particular site is a content farm (b) and therefore the number of sites that people seem to agree are farms aren't more than a dozen.
- tnorthcutt 16y agoDoes anybody else feel that Google has been figured out now, and that they are now just applying patches to a system that is fundamentally broken (in the same way Excite and AV did)? I don't. I think they do a much better job of actively improving their offering. No matter what the system is, someone will always try to game it. Even if the people running the system are very smart and try very hard, the smart gamers will (briefly) succeed. That's ok, though - it pushes the people running the system to work harder to improve it. It really is in Google's best interest (IMO) to return great search results, so I think they'll continue to improve what they do.
- alanthonyc 16y ago
- JamesDB 16y agoHow will this impact sites like Rotten Tomatoes? They have little original content, but are still a useful source.