6 ms·

Spam sites in Google News

153 points by gnewssheriff 12y ago

eli 12y ago
Google News has always given off this vibe like it's being administered by someone as their side project. Very curious for one of the most popular news websites in the world.
- tymekpavel 12y ago
  Without going into too much detail, this is essentially true. The site is built on top of algorithms that were developed years ago, and there are some runbooks to help SREs keep the site running. However, there isn't much investment going on. Occasionally they'll have one engineer or an intern add a feature, but it's not permanently staffed.
  - Scoundreller 12y ago
    Would this change if they started running ads on it? I guess the question is, is the lack of investment because it's a cost-centre, or for some other reason?
    - larrys 12y ago
      One of the reasons is that Google doesn't seem to want to hire the "b" team. And the "a" team I'm guessing doesn't want to work in a backwater place with no glory like "google news". You know they only hire the best and the brightest and all of that. I'm sure even the person who gets hired as a janitor is a cut well above the average janitor. [1] The funny part is there are plenty of people who would die to work at Google on anything. But they aren't the type (total conjecture here btw) that would ever pass the google interview process. [1] I've always thought this was an interesting paradox. That is someone gets a job as a maid in the White House and works near the President let's say. So she/he must have something going for them to get that type of job. But yet they are still a maid in the white house. You would think if you are able to land that job you would have risen above that job.
      - username223 12y ago
        > a backwater place with no glory like "google news". I know they have lots of people working in their "social" wasteland (G+, hangouts, etc.). I can only assume they already gather most of the "news" tracking information by other means, so GN isn't worth supporting.
      - eli 12y ago
        I guess I don't get why it's a backwater. They have insanely high traffic numbers. They have interesting problems of surfacing relevant news and grouping stories around topics. Seems like there are tons of cool things you could do with the site, if you were so inclined.
      - UrMomReadsHN 12y ago
        >You would think if you are able to land that job you would have risen above that job. What an awful thing to say. Some people like doing that type of work. There is nothing wrong with that. I don't think our society is destined for greatness when we devalue important work. As an aside, my aunt did a lot of jobs and always went back to being a cleaner because thats what she likes to do.
    - nl 12y ago
      They can't run ads against it unless they manage to work out some arrangement with the News Publishers. Technically they might have the legal right to do it, but the Google/News publisher relationship is already strained enough, and Google wouldn't want to strain it more.
      - eli 12y ago
        I don't see why that would be true. Google News is just links to other sites like any other google search, except for some wire articles that they are actually paying for.
- r721 12y ago
  I use the number of blog posts a year in a Google product's blog as an estimate for its importance for Google: 2014: 3 2013: 3 2012: 8 http://googlenewsblog.blogspot.com http://googlenewsblog.blogspot.com So not very important in last two years as it seems.
  - mnx 12y ago
    for comparison: Gmail: 2014: 21 2013: 25 2012: 35 2011: 84 Reader: 2013: 2 2012: 0 (!) 2011: 4 2010: 16
- marincounty 12y ago
  Yea, I understand what Google is trying to do with their news site, but it'a not working--for me--in the Bay Area. They have money to burn/waste/spoil Geniuses? Why not hire the best reporters and put together news sites for every country on the planet? Old fashioned reporting? Stop leaching off struggling news papers in order to obtain your content? The internet has destroyed so many good newspapers. Why not reverse the trend and bring back good journalism? Or just, give some of the struggling newspapers cold hard cash--in the form of a grant, before we lose them all? Or, they just become pet/vanity projects for rich tech guys?
kumarski 12y ago
Thank you for doing this.
danso 12y ago
I used to work on a news site and the process to get listed on Google News was very manual, at least compared to how you think the rest of Google operates. You had to have a physical address and phone number (though I don't think the two attributes are actually tested, i.e. sending a postcard via snailmail). You also had to configure your pages for the spider so that if your site delivers wire content (e.g. Reuters, AP) or does aggregation, the spider would ignore those pages, and only pick up your original news. The first time we submitted, the response we got seemed to indicate that someone had looked over our sitemap and saw aggregated news/wire reports, and so we had to resubmit. The process took a few weeks, IIRC. That said, bigger news sites were probably whitelisted...it seems like the whitelist could be done algorithmically...but the number of new authoritative news sites, in theory, changes so slowly, that there's probably not really a need to do that.
- walshemj 12y ago
  actually no I used to work at one of the big publishers we did not get our properties listed automatically.
- eli 12y ago
  You left out the most bizarre part of the process -- the insane constraints on URLs. For example: "The URL for each article must contain a unique number consisting of at least three digits. For example, we can't crawl an article with this URL: http://www.google.com/news/article23.html http://www.google.com/news/article23.html. We can, however, crawl an article with this URL: http://www.google.com/news/article234.html http://www.google.com/news/article234.html. " (Source: https://support.google.com/news/publisher/answer/68323?hl=en https://support.google.com/news/publisher/answer/68323?hl=en)
  - declan 12y ago
    When I worked at CNET, we were (and still are) indexed by Google News despite not having three-digit numbers in URLs. That's because your excerpt from Google News' guidelines left off a very important addendum: "Please note that this rule is waived with News sitemaps."
tokenadult 12y ago
There is a lot of good information in the post submitted here, which should be actionable for Google if Google cares about the quality of news results. I'll note for the record that I have been using Google News as a news aggregator since the beginning of its existence, and since I almost use Google News in a logged-in condition, Google News responds to the way I have trained it about my news interests by mostly showing me stories from established news organizations all around the world, and not from spammy linkfarms. My current gripe about Google News is that it pushes far too many low-quality or too local news outlets into the Editor's Picks section of my view of Google News, and even if I click repeatedly as those sources display "Personalize this news source" to display it rarely, the same podunk local TV station or trade magazine website will keep displaying in that section over and over and over. (I have already complained to Google through Google News feedback channels about this behavior. It should be possible to mark a source as NEVER appearing in the "Editor's Picks" section and have that selection be implemented until the user affirmatively turns it off.) On the whole, I like Google News. But for sure if a site has eyeballs, spammers will try to grab those eyeballs, so the price of freedom from spam is eternal vigilance.
Animats 12y ago
One of the sites mentioned, "wtnnews.com" ("West Texas News"), is a straightforward content farm running Google Adsense ads. Here's their home site, "wtnmedia.com", "A global business-to-business media company where you can forge high level client relationships through the web, print, and events." Their advertising kit shows a base ad rate is $1,295 for 7,500 impressions of a banner ad, for a Cost per Thousand of $171. Nobody would pay that; typical CPTs today are around $1. So that's not their revenue source. The use of the business addresses of other businesses is unusual. That's identity theft. Our SiteTruth system checks out the address of the business found on the web site, and it brings up StreetView pictures of auto repair shops for some of those sites. Lots of sites don't give an address, or use some mail drop, but use of fake street addresses is rare. It tends to draw tax collector and law enforcement attention. Google's anti-spam efforts have never been very aggressive. They filter out some of the worst offenders, but don't try hard to get rid of bottom-feeder sites like these. If Google didn't put spam sites in news results, how would they get revenue? Google News itself has no ads, and many of the top news outlets have their own advertising systems that don't involve Google.
- a3n 12y ago
  "... where you can forge high level client relationships ..." Great double entendre.
graupel 12y ago
The funny thing is that as a large publisher, it's not links on news.google.com that we really care about, it's landing in universal Google search one-boxes as a result of being indexed in Google News - that's where the traffic really comes from.
- Destitute 12y ago
  Yea, there's a lot of sites that take advantage of this like The Christian Post who target high search keywords for their "news" that has no relevance to the actual term. Search for "Stream NFL games" and "Watch NFL football online free" and other variants and see them pop up for everything with no mention of actually streaming games online (obviously). See: http://www.christianpost.com/news/carolina-panthers-vs-seattle-seahawks-live-stream-free-watch-nfl-2015-playoffs-football-online-fox-tv-start-time-preview-132376/ http://www.christianpost.com/news/carolina-panthers-vs-seatt...
  - notadoc 12y ago
    Many very top sites do this with any popular news term or keyword, its incentive to create content farms. Madness.
everydaypanos 12y ago
+1 for exposing the fake writer profiles. Very amusing!
alternatives 12y ago
We need alternatives to Google. We can't have one company deciding what gets seen on the WWW. It is a great search engine, Google changed the world. But now we need alternatives. We can't have their editorial staff dictate what should and should not be seen on the WWW. If you look closely at the top search results, they're mostly big spenders in Adwords. Ebay, Amazon, Expedia, TripAdvisor, Yelp, Answers.com and several others. It's blatant conflict of interests. There is no way those search results are "organic". Basically the WWW has become controlled and curated by Google and every site that gets seen must conform or be destroyed. We need alternatives to Google. We need them urgently.
- richthegeek 12y ago
  It's quite possible that the results are "organic" in the naive definition by following this process: 1. {site} starts at rank 1 due to low visitors/connectivity 2. {site} pays for adwords driving new visitors 3. visitors link to the site 4. rank of the site increases allowing increase adwords spend 5. goto 2 The suggestion that Google artificially inflates the rank of big spenders is maybe conflating correlation and causation. On the other hand, it's not impossible or even that difficult to believe.
- username223 12y ago
  > We need alternatives to Google. We need them urgently. Thanks to the 80/20 rule, we have them. Google is still unmatched for edge cases in maps and search, but something else works 80% of the time, and Google is spending 80% of the effort for that last 20%.
  - ooOOoo 12y ago
    Isn't it the opposite? Google maps works 80% of the time, but for the 20% remaining, you need to look at more specialized maps. For example look at https://www.google.com/maps?q=lamma+island https://www.google.com/maps?q=lamma+island : almost nothing, just of few paths. Now look at http://www.map.gov.hk/ http://www.map.gov.hk/ : all paths and houses in Lamma are precisely described!
- runn1ng 12y ago
thezach 12y ago
I reached out to Emily Linnert, one of the real journalist who's photo is being used on one of the sites and shes replied. Her (or her station) is taking legal action. https://twitter.com/TheZach/status/554450915319492608 https://twitter.com/TheZach/status/554450915319492608
jmarbach 12y ago
I too have found some serious quality control issues with Google News. Some news sites are redirecting their mobile viewers (knowingly or not) to the app store instead of giving them the article that they clicked on. I wrote about this experience here: http://jmarbach.com/google-news-growth-hack-exposed http://jmarbach.com/google-news-growth-hack-exposed
stevenh 12y ago
I noticed that one of the scam sites is currently listed for sale on Flippa. On their listing, they even openly boast that the best way to use the site is to use its Google News status to scam Google's search results: https://flippa.com/3500230-google-news-approved-site-with-143-951-uniques-mo-making-2-132-mo https://flippa.com/3500230-google-news-approved-site-with-14... Here's a mirror, in case it gets modified or deleted: https://archive.today/Jzubr https://archive.today/Jzubr
- gnewssheriff 12y ago
  Update: Site has been cataloged and added: http://googlenewsspam.com/index_.php http://googlenewsspam.com/index_.php Thanks for your help.
zaroth 12y ago
Where's Matt Cutts when you need him? [1] [1] - https://www.mattcutts.com/blog/on-leave/ https://www.mattcutts.com/blog/on-leave/

153 points by gnewssheriff 12y ago

eli 12y ago

Google News has always given off this vibe like it's being administered by someone as their side project. Very curious for one of the most popular news websites in the world.

tymekpavel 12y ago
Without going into too much detail, this is essentially true. The site is built on top of algorithms that were developed years ago, and there are some runbooks to help SREs keep the site running. However, there isn't much investment going on. Occasionally they'll have one engineer or an intern add a feature, but it's not permanently staffed.
- Scoundreller 12y ago
  Would this change if they started running ads on it? I guess the question is, is the lack of investment because it's a cost-centre, or for some other reason?
  - larrys 12y ago
    One of the reasons is that Google doesn't seem to want to hire the "b" team. And the "a" team I'm guessing doesn't want to work in a backwater place with no glory like "google news". You know they only hire the best and the brightest and all of that. I'm sure even the person who gets hired as a janitor is a cut well above the average janitor. [1] The funny part is there are plenty of people who would die to work at Google on anything. But they aren't the type (total conjecture here btw) that would ever pass the google interview process. [1] I've always thought this was an interesting paradox. That is someone gets a job as a maid in the White House and works near the President let's say. So she/he must have something going for them to get that type of job. But yet they are still a maid in the white house. You would think if you are able to land that job you would have risen above that job.
    - username223 12y ago
      > a backwater place with no glory like "google news". I know they have lots of people working in their "social" wasteland (G+, hangouts, etc.). I can only assume they already gather most of the "news" tracking information by other means, so GN isn't worth supporting.
    - eli 12y ago
      I guess I don't get why it's a backwater. They have insanely high traffic numbers. They have interesting problems of surfacing relevant news and grouping stories around topics. Seems like there are tons of cool things you could do with the site, if you were so inclined.
    - UrMomReadsHN 12y ago
      >You would think if you are able to land that job you would have risen above that job. What an awful thing to say. Some people like doing that type of work. There is nothing wrong with that. I don't think our society is destined for greatness when we devalue important work. As an aside, my aunt did a lot of jobs and always went back to being a cleaner because thats what she likes to do.
  - nl 12y ago
    They can't run ads against it unless they manage to work out some arrangement with the News Publishers. Technically they might have the legal right to do it, but the Google/News publisher relationship is already strained enough, and Google wouldn't want to strain it more.
    - eli 12y ago
      I don't see why that would be true. Google News is just links to other sites like any other google search, except for some wire articles that they are actually paying for.
r721 12y ago
I use the number of blog posts a year in a Google product's blog as an estimate for its importance for Google: 2014: 3 2013: 3 2012: 8 http://googlenewsblog.blogspot.com http://googlenewsblog.blogspot.com So not very important in last two years as it seems.
- mnx 12y ago
  for comparison: Gmail: 2014: 21 2013: 25 2012: 35 2011: 84 Reader: 2013: 2 2012: 0 (!) 2011: 4 2010: 16
marincounty 12y ago
Yea, I understand what Google is trying to do with their news site, but it'a not working--for me--in the Bay Area. They have money to burn/waste/spoil Geniuses? Why not hire the best reporters and put together news sites for every country on the planet? Old fashioned reporting? Stop leaching off struggling news papers in order to obtain your content? The internet has destroyed so many good newspapers. Why not reverse the trend and bring back good journalism? Or just, give some of the struggling newspapers cold hard cash--in the form of a grant, before we lose them all? Or, they just become pet/vanity projects for rich tech guys?

kumarski 12y ago

Thank you for doing this.

danso 12y ago

I used to work on a news site and the process to get listed on Google News was very manual, at least compared to how you think the rest of Google operates. You had to have a physical address and phone number (though I don't think the two attributes are actually tested, i.e. sending a postcard via snailmail). You also had to configure your pages for the spider so that if your site delivers wire content (e.g. Reuters, AP) or does aggregation, the spider would ignore those pages, and only pick up your original news. The first time we submitted, the response we got seemed to indicate that someone had looked over our sitemap and saw aggregated news/wire reports, and so we had to resubmit. The process took a few weeks, IIRC. That said, bigger news sites were probably whitelisted...it seems like the whitelist could be done algorithmically...but the number of new authoritative news sites, in theory, changes so slowly, that there's probably not really a need to do that.

walshemj 12y ago
actually no I used to work at one of the big publishers we did not get our properties listed automatically.
eli 12y ago
You left out the most bizarre part of the process -- the insane constraints on URLs. For example: "The URL for each article must contain a unique number consisting of at least three digits. For example, we can't crawl an article with this URL: http://www.google.com/news/article23.html http://www.google.com/news/article23.html. We can, however, crawl an article with this URL: http://www.google.com/news/article234.html http://www.google.com/news/article234.html. " (Source: https://support.google.com/news/publisher/answer/68323?hl=en https://support.google.com/news/publisher/answer/68323?hl=en)
- declan 12y ago
  When I worked at CNET, we were (and still are) indexed by Google News despite not having three-digit numbers in URLs. That's because your excerpt from Google News' guidelines left off a very important addendum: "Please note that this rule is waived with News sitemaps."

tokenadult 12y ago

There is a lot of good information in the post submitted here, which should be actionable for Google if Google cares about the quality of news results. I'll note for the record that I have been using Google News as a news aggregator since the beginning of its existence, and since I almost use Google News in a logged-in condition, Google News responds to the way I have trained it about my news interests by mostly showing me stories from established news organizations all around the world, and not from spammy linkfarms. My current gripe about Google News is that it pushes far too many low-quality or too local news outlets into the Editor's Picks section of my view of Google News, and even if I click repeatedly as those sources display "Personalize this news source" to display it rarely, the same podunk local TV station or trade magazine website will keep displaying in that section over and over and over. (I have already complained to Google through Google News feedback channels about this behavior. It should be possible to mark a source as NEVER appearing in the "Editor's Picks" section and have that selection be implemented until the user affirmatively turns it off.) On the whole, I like Google News. But for sure if a site has eyeballs, spammers will try to grab those eyeballs, so the price of freedom from spam is eternal vigilance.

Animats 12y ago

One of the sites mentioned, "wtnnews.com" ("West Texas News"), is a straightforward content farm running Google Adsense ads. Here's their home site, "wtnmedia.com", "A global business-to-business media company where you can forge high level client relationships through the web, print, and events." Their advertising kit shows a base ad rate is $1,295 for 7,500 impressions of a banner ad, for a Cost per Thousand of $171. Nobody would pay that; typical CPTs today are around $1. So that's not their revenue source. The use of the business addresses of other businesses is unusual. That's identity theft. Our SiteTruth system checks out the address of the business found on the web site, and it brings up StreetView pictures of auto repair shops for some of those sites. Lots of sites don't give an address, or use some mail drop, but use of fake street addresses is rare. It tends to draw tax collector and law enforcement attention. Google's anti-spam efforts have never been very aggressive. They filter out some of the worst offenders, but don't try hard to get rid of bottom-feeder sites like these. If Google didn't put spam sites in news results, how would they get revenue? Google News itself has no ads, and many of the top news outlets have their own advertising systems that don't involve Google.

a3n 12y ago
"... where you can forge high level client relationships ..." Great double entendre.

graupel 12y ago

The funny thing is that as a large publisher, it's not links on news.google.com that we really care about, it's landing in universal Google search one-boxes as a result of being indexed in Google News - that's where the traffic really comes from.

Destitute 12y ago
Yea, there's a lot of sites that take advantage of this like The Christian Post who target high search keywords for their "news" that has no relevance to the actual term. Search for "Stream NFL games" and "Watch NFL football online free" and other variants and see them pop up for everything with no mention of actually streaming games online (obviously). See: http://www.christianpost.com/news/carolina-panthers-vs-seattle-seahawks-live-stream-free-watch-nfl-2015-playoffs-football-online-fox-tv-start-time-preview-132376/ http://www.christianpost.com/news/carolina-panthers-vs-seatt...
- notadoc 12y ago
  Many very top sites do this with any popular news term or keyword, its incentive to create content farms. Madness.

everydaypanos 12y ago

+1 for exposing the fake writer profiles. Very amusing!

alternatives 12y ago

We need alternatives to Google. We can't have one company deciding what gets seen on the WWW. It is a great search engine, Google changed the world. But now we need alternatives. We can't have their editorial staff dictate what should and should not be seen on the WWW. If you look closely at the top search results, they're mostly big spenders in Adwords. Ebay, Amazon, Expedia, TripAdvisor, Yelp, Answers.com and several others. It's blatant conflict of interests. There is no way those search results are "organic". Basically the WWW has become controlled and curated by Google and every site that gets seen must conform or be destroyed. We need alternatives to Google. We need them urgently.

richthegeek 12y ago
It's quite possible that the results are "organic" in the naive definition by following this process: 1. {site} starts at rank 1 due to low visitors/connectivity 2. {site} pays for adwords driving new visitors 3. visitors link to the site 4. rank of the site increases allowing increase adwords spend 5. goto 2 The suggestion that Google artificially inflates the rank of big spenders is maybe conflating correlation and causation. On the other hand, it's not impossible or even that difficult to believe.
username223 12y ago
> We need alternatives to Google. We need them urgently. Thanks to the 80/20 rule, we have them. Google is still unmatched for edge cases in maps and search, but something else works 80% of the time, and Google is spending 80% of the effort for that last 20%.
- ooOOoo 12y ago
  Isn't it the opposite? Google maps works 80% of the time, but for the 20% remaining, you need to look at more specialized maps. For example look at https://www.google.com/maps?q=lamma+island https://www.google.com/maps?q=lamma+island : almost nothing, just of few paths. Now look at http://www.map.gov.hk/ http://www.map.gov.hk/ : all paths and houses in Lamma are precisely described!
runn1ng 12y ago

thezach 12y ago

I reached out to Emily Linnert, one of the real journalist who's photo is being used on one of the sites and shes replied. Her (or her station) is taking legal action. https://twitter.com/TheZach/status/554450915319492608 https://twitter.com/TheZach/status/554450915319492608

jmarbach 12y ago

I too have found some serious quality control issues with Google News. Some news sites are redirecting their mobile viewers (knowingly or not) to the app store instead of giving them the article that they clicked on. I wrote about this experience here: http://jmarbach.com/google-news-growth-hack-exposed http://jmarbach.com/google-news-growth-hack-exposed

stevenh 12y ago

I noticed that one of the scam sites is currently listed for sale on Flippa. On their listing, they even openly boast that the best way to use the site is to use its Google News status to scam Google's search results: https://flippa.com/3500230-google-news-approved-site-with-143-951-uniques-mo-making-2-132-mo https://flippa.com/3500230-google-news-approved-site-with-14... Here's a mirror, in case it gets modified or deleted: https://archive.today/Jzubr https://archive.today/Jzubr

gnewssheriff 12y ago
Update: Site has been cataloged and added: http://googlenewsspam.com/index_.php http://googlenewsspam.com/index_.php Thanks for your help.

zaroth 12y ago

Where's Matt Cutts when you need him? [1] [1] - https://www.mattcutts.com/blog/on-leave/ https://www.mattcutts.com/blog/on-leave/