7 ms·

Belgian newspapers: Google blocking us on searches

39 points by robtoo 15y ago

itcmcgrath 15y ago
The only comment that comes to mind: Don't bite the hand that feeds you. The newspapers really didn't think that one through, did they...
- shrikant 15y ago
  I suppose to extend that analogy: the newspapers didn't believe they were biting the hand that was feeding. They saw it more as cutting off the other hand that pushed their heads towards the food. Google is saying both hands are equivalent, but the newspapers are saying they want one hand to feed them, and the other to keep away. I can buy that, actually. Sure, I don't agree with the initial ruling, but IMHO de-indexing the newspapers from the search engine was unnecessarily retaliatory. I can understand why Google did it, but having made their point, I think they should gracefully back down claiming ignorance. This is a clear misuse of their 'monopoly' on web search for benefit in other areas. EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384 http://news.ycombinator.com/item?id=2776384)
  - yock 15y ago
    Unnecessarily retalitory? I wouldn't be so sure. Don't forget that Google search results include selected content from indexed websites. The index itself actually contains all content, which Google may have believed to be a litigation risk given the vagueness of the removal request.
  - turtle4 15y ago
    Google supplies the tools for the newspaper to have marked explicitly, using technology, the pages they want included/excluded from search, with no ambiguity. Instead, they chose to use vague, court language, with threat of a fine per link, to force google's hand. Personally, I don't know why google -wouldn't- drop them, permanently. There is no monopoly on web search. Search works for everyone, specifically because there are established protocols and conventions that allow all these systems to work together. If your company chooses to ignore these conventions, I don't think it is unreasonable that you get ignored. You don't have a -right- to be included in a web search. The reason google includes you (or any search engine) is because you are making a trade, they get better results which they can monetize via ads, you get more traffic, which you monetize through ads. If you opt out, why shouldn't they? What's in it for them? They aren't running a charity, they are running a business.
    - ToadyCensored 15y ago
      Your post demonstrates a misunderstanding over what the lawsuit was about. Publishers weren't suing to make Google stop linking to them. Google News was displaying full excerpts from copyright articles, sometimes entire paragraphs, and on top of that making money off that content through the display of ads. It was a copyright infringement lawsuit. There is no monopoly on web search. Google has over 65% market share in search. Second is Yahoo with only 16%.
  - muuh-gnu 15y ago
    > they want one hand to feed them, and the other to keep away. They dont seem to be in a negotiating position to express "wishes" which then somehow have to be granted in the exact way they want them.
  - deleted 15y ago
    [deleted]
  - arhughes 15y ago
    From http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20appeal%20Google_5May2011.pdf http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20app... Section V: "Orders Google to remove from Google.be and Google.com sites, more specifically from the "cached" links on "Google Web" and from the "Google News" service, all articles..." Depending on your interpretation, they were supposed to remove the newspapers from Google News and remove the "cached" links from Google Search results.
  - melvinmt 15y ago
    There's a € 25.000 euro fine on every violation of the court order. Google doesn't want to take a chance to leave room for even the smallest amount of ambiguity, so that's why they've removed all links to those newspapers.
    - Toady 15y ago
      The lawsuit was pertaining to Google News, not the general search engine. Google removed the links as punishment for challenging them in court, to show the content-owners that Google controls their revenue streams.
  - ToadyCensored 15y ago
    * EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384 http://news.ycombinator.com/item?id=2776384) * That's in reference to the Google cache of lead paragraphs and other article excerpts. The lawsuit was not about general links in the search engine. Publishers sued because Google News was displaying lead paragraphs and other copyrighted writing in the aggregator. If you started your own newspaper that copied all the headlines and lead paragraphs from the New York Times, it wouldn't be long before the New York Times sent you a legal letter.
- bonch 15y ago
  This is Google playing politics. The lawsuit was over Google News copying the lead paragraph from other people's articles, and now Google is strong-arming them and showing them how much power they have--mess with Google, and they remove you from their search engine.
slater 15y ago
Anyone have a copy of the ruling? Might just be the judge(s) misinterpreting what the newspapers were asking for?
- roel_v 15y ago
  Ruling (in French): http://www.ie-forum.nl/backoffice/uploads/file/IEF%20Hof%20van%20Beroep%20Brussel%205%20mei%202011,%20R_nr_%202011_2999,%20817%20%28Google%20INC_%20tegen%20Copiepresse,%20Societe%20de%20Droit%20d%27auteur%20des%20journalistes%20%28SAJ%29%20en%20Assucopie%29.pdf http://www.ie-forum.nl/backoffice/uploads/file/IEF%20Hof%20v... Can't copy and paste from there; relevant passage: "...orders Google to removing the sites google.be and google.com, more precisely the links (in cache) visible through Google Web and the service Google News (...) of Belgian Francophone and German-language publishers represented by Copiepresse." So yes, Google is interpreting the ruling correctly, there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much or if the judge didn't understand what he was doing; I didn't read the whole ruling.
  - masklinn 15y ago
    > there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much I think theoretically it would be possible if and only if they specialized Google's crawler and rendering to only render a link (no preview, no cache link) for these sites and these sites only. Why they'd bother doing that is anyone's guess.
  - Toady 15y ago
    The lawsuit was over the use of lead paragraphs and other article excerpts, specifically in the Google News aggregator, not plain links in the general search engine.
    - roel_v 15y ago
      Yes of course, but the ruling was broader than that, that is the whole point of contention.
dexen 15y ago
The newspapers went to court asking for broad relief (unlisting from google results), instead of putting in place proper META tags and /robots.txt to direct what may be, and what may not be indexed and cached. The court handed them just as much of the proverbial rope as they asked for. Google's web crawlers asked the WWW servers to GET, were served with HTTP 200 OK, found no prohibition in /robots.txt and so followed the standard protocol. Note Google Web Search and Google News use separate bots (Googlebot and Googlebot-News) to give website owner full control.
- reso 15y ago
  I can't decide whether I find things like this hilarious or disturbing. They could have fixed it with engineering, but they were either so uninformed or incompetent that they tried to fix it with a lawsuit.
  - pavel_lishin 15y ago
    Or maybe one incompetent overrode them all. Manager: Evans, get in here! Why are our news articles all over The Google? Evans: Um, sir, they index our - Manager: I don't want to hear your technical mumbo jumbo! Get legal on this to get it fixed YESTERDAY! Evans: There's no need, it's literally a 5-minute fix, I could just add an entry to robots.txt while the coffee is brewing... Manager: What did I just say about mumbo jumbo? Get out of here, and send in Stevens from Legal on your way.
    - Toady 15y ago
      Again, the lawsuit was over Google News' display of lead paragraphs and other excerpts from articles. It didn't pertain to the general search engine.
      - pavel_lishin 15y ago
        The lawsuit may not have, but the ruling certainly and explicitly does.
lenni 15y ago
Maybe I just don't get it, but why do they want to be included in web search but not in Google News? Their actions seem bizarre. Both are noindex-able separately, correct?
- dexen 15y ago
  tl;dr: Yes they are indexed separately, and yes it's up to site owner to decide which content may be indexed for Google News -- via proper META tags and robots.txt. There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1] Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here. To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit! http://googlewebmastercentral.blogspot.com/2009/12/new-user-agent-for-news.html http://googlewebmastercentral.blogspot.com/2009/12/new-user-... ---- [1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.
- ChuckMcM 15y ago
  Short answer, one looks like a newspaper. The longer answer is that the newspapers are struggling financially, Google is not struggling, when snippets of content appear on the Google news page the newspapers believe they are 'improving' the Google brand without being compensated. They sued to have either Google pay them, or remove them from the News site. What they fail to realize (and Google no doubt knows) is that if you aren't listed in Google you're not going to get nearly as much traffic (hence 'profit'). So Google apparently took a literal interpretation of the order and struck them from the books as it were, complete removal from the index. So the newspapers have to make the following evaluation, 'what is it worth to us to be in Google's search results?' And is that value more or less than what we think they owe us for having our results on their news page? Google clearly understands that its value for them to be in the search results and I believe they would say off the record that the value of being in the results is worth more than the ad revenue they get from the occasional story that makes it to the front page of Google news. Sort of extortion, sort of natural consequences. Humorous though.
bugsy 15y ago
OK, it says they filed a lawsuit against Google because they didn't want Google to LINK to their articles without paying a fee for linking. They won the lawsuit in court and so Google removed links to their articles from their search index, as they requested, and now they are mad about it. Why is the story "Google blocking" them. Google's only doing what a court ordered after the papers sued asking for that.
- pavel_lishin 15y ago
  As I understand it, they wanted Google News to stop linking to their articles; not the regular search results.
  - dexen 15y ago
    From court order, via http://www.chillingeffects.org/notice.cgi?sID=2160 http://www.chillingeffects.org/notice.cgi?sID=2160 > Order the defendant to withdraw the articles (...) from all their sites (Google News and "cache" Google or any other name (...) The other articles linked from http://slashdot.org/story/06/09/18/115203/Google-News-Removes-Belgian-Newspaper http://slashdot.org/story/06/09/18/115203/Google-News-Remove... are down now, but I believe the original lawsuit was in that very spirit: remove their articles from all Google's websites. Not just from Google News. Logical argumentation aside, Google's in no position to second-guess a court order ;-) Why did Associated Press put such a negative spin to the story? We can only guess...
- Toady 15y ago
  They sued because Google News was using lead paragraphs from their articles. Google is most definitely block them, because the court's decision didn't pertain to the general search engine. In effect, Google is punishing these content-owners for challenging Google's use of their content in its service.
Luc 15y ago
This is not a case of technical ignorance, though it's hilarious to pretend it is. The French-language newspaper association wants to receive money from Google (for the ads displayed next to their content) and figured it would be easier to pressure them through the courts than through other means.
- ToadyCensored 15y ago
  The lawsuit was over the use of copyrighted excerpts from articles in the Google News aggregator and cache, such as lead paragraphs. Google is trying to pretend that the court ruled all hyperlinks to be illegal. Google was defiant before over this ruling when it ignored an order to publish the results of the verdict on its website, so it's not far-fetched to believe Google is once again throwing its weight around and reminding people of the power it wields over the livelihoods of content creators.
franze 15y ago
User-agent: Googlebot Disallow: User-agent: Googlebot-News Disallow: /
Bootvis 15y ago
The papers are back according to this news article: http://babelfish.yahoo.com/translate_url?doit=done&tt=url&intl=1&fr=bf-home&trurl=http%3A%2F%2Fwww.nu.nl%2Finternet%2F2568534%2Fbelgische-kranten-weer-google.html&lp=nl_en&btnTrUrl=Translate http://babelfish.yahoo.com/translate_url?doit=done&tt=ur... They also hint that the complaints have been lifted by the papers.
- feb 15y ago
  Yes, they are back in the index: http://www.pcmag.com/article2/0,2817,2388635,00.asp http://www.pcmag.com/article2/0,2817,2388635,00.asp And in a simple test, it looks to work too: http://www.google.com/search?q=copiepress+lalibre+critique http://www.google.com/search?q=copiepress+lalibre+critique The second link (lesoir.be) is one of the newspapers.

39 points by robtoo 15y ago

The only comment that comes to mind: Don't bite the hand that feeds you. The newspapers really didn't think that one through, did they...

shrikant 15y ago
I suppose to extend that analogy: the newspapers didn't believe they were biting the hand that was feeding. They saw it more as cutting off the other hand that pushed their heads towards the food. Google is saying both hands are equivalent, but the newspapers are saying they want one hand to feed them, and the other to keep away. I can buy that, actually. Sure, I don't agree with the initial ruling, but IMHO de-indexing the newspapers from the search engine was unnecessarily retaliatory. I can understand why Google did it, but having made their point, I think they should gracefully back down claiming ignorance. This is a clear misuse of their 'monopoly' on web search for benefit in other areas. EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384 http://news.ycombinator.com/item?id=2776384)
- yock 15y ago
  Unnecessarily retalitory? I wouldn't be so sure. Don't forget that Google search results include selected content from indexed websites. The index itself actually contains all content, which Google may have believed to be a litigation risk given the vagueness of the removal request.
- turtle4 15y ago
  Google supplies the tools for the newspaper to have marked explicitly, using technology, the pages they want included/excluded from search, with no ambiguity. Instead, they chose to use vague, court language, with threat of a fine per link, to force google's hand. Personally, I don't know why google -wouldn't- drop them, permanently. There is no monopoly on web search. Search works for everyone, specifically because there are established protocols and conventions that allow all these systems to work together. If your company chooses to ignore these conventions, I don't think it is unreasonable that you get ignored. You don't have a -right- to be included in a web search. The reason google includes you (or any search engine) is because you are making a trade, they get better results which they can monetize via ads, you get more traffic, which you monetize through ads. If you opt out, why shouldn't they? What's in it for them? They aren't running a charity, they are running a business.
  - ToadyCensored 15y ago
    Your post demonstrates a misunderstanding over what the lawsuit was about. Publishers weren't suing to make Google stop linking to them. Google News was displaying full excerpts from copyright articles, sometimes entire paragraphs, and on top of that making money off that content through the display of ads. It was a copyright infringement lawsuit. There is no monopoly on web search. Google has over 65% market share in search. Second is Yahoo with only 16%.
- muuh-gnu 15y ago
  > they want one hand to feed them, and the other to keep away. They dont seem to be in a negotiating position to express "wishes" which then somehow have to be granted in the exact way they want them.
- deleted 15y ago
  [deleted]
- arhughes 15y ago
  From http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20appeal%20Google_5May2011.pdf http://www.copiepresse.be/pdf/Copiepresse%20-%20ruling%20app... Section V: "Orders Google to remove from Google.be and Google.com sites, more specifically from the "cached" links on "Google Web" and from the "Google News" service, all articles..." Depending on your interpretation, they were supposed to remove the newspapers from Google News and remove the "cached" links from Google Search results.
- melvinmt 15y ago
  There's a € 25.000 euro fine on every violation of the court order. Google doesn't want to take a chance to leave room for even the smallest amount of ambiguity, so that's why they've removed all links to those newspapers.
  - Toady 15y ago
    The lawsuit was pertaining to Google News, not the general search engine. Google removed the links as punishment for challenging them in court, to show the content-owners that Google controls their revenue streams.
- ToadyCensored 15y ago
  * EDIT: Oh good, my Google fanboi-ness is not in vain. The court ruling actually ordered that links be removed from "Google Web [sic]" as well. Oh well, the joke's on the newspapers then, as it appears they asked for it! (see roel_v's comment: http://news.ycombinator.com/item?id=2776384 http://news.ycombinator.com/item?id=2776384) * That's in reference to the Google cache of lead paragraphs and other article excerpts. The lawsuit was not about general links in the search engine. Publishers sued because Google News was displaying lead paragraphs and other copyrighted writing in the aggregator. If you started your own newspaper that copied all the headlines and lead paragraphs from the New York Times, it wouldn't be long before the New York Times sent you a legal letter.
bonch 15y ago
This is Google playing politics. The lawsuit was over Google News copying the lead paragraph from other people's articles, and now Google is strong-arming them and showing them how much power they have--mess with Google, and they remove you from their search engine.

slater 15y ago

Anyone have a copy of the ruling? Might just be the judge(s) misinterpreting what the newspapers were asking for?

roel_v 15y ago
Ruling (in French): http://www.ie-forum.nl/backoffice/uploads/file/IEF%20Hof%20van%20Beroep%20Brussel%205%20mei%202011,%20R_nr_%202011_2999,%20817%20%28Google%20INC_%20tegen%20Copiepresse,%20Societe%20de%20Droit%20d%27auteur%20des%20journalistes%20%28SAJ%29%20en%20Assucopie%29.pdf http://www.ie-forum.nl/backoffice/uploads/file/IEF%20Hof%20v... Can't copy and paste from there; relevant passage: "...orders Google to removing the sites google.be and google.com, more precisely the links (in cache) visible through Google Web and the service Google News (...) of Belgian Francophone and German-language publishers represented by Copiepresse." So yes, Google is interpreting the ruling correctly, there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much or if the judge didn't understand what he was doing; I didn't read the whole ruling.
- masklinn 15y ago
  > there is no way to interpret it in such a way that the regular google.com shouldn't be blocked. I don't know if the newspapers asked for too much I think theoretically it would be possible if and only if they specialized Google's crawler and rendering to only render a link (no preview, no cache link) for these sites and these sites only. Why they'd bother doing that is anyone's guess.
- Toady 15y ago
  The lawsuit was over the use of lead paragraphs and other article excerpts, specifically in the Google News aggregator, not plain links in the general search engine.
  - roel_v 15y ago
    Yes of course, but the ruling was broader than that, that is the whole point of contention.

dexen 15y ago

The newspapers went to court asking for broad relief (unlisting from google results), instead of putting in place proper META tags and /robots.txt to direct what may be, and what may not be indexed and cached. The court handed them just as much of the proverbial rope as they asked for. Google's web crawlers asked the WWW servers to GET, were served with HTTP 200 OK, found no prohibition in /robots.txt and so followed the standard protocol. Note Google Web Search and Google News use separate bots (Googlebot and Googlebot-News) to give website owner full control.

reso 15y ago
I can't decide whether I find things like this hilarious or disturbing. They could have fixed it with engineering, but they were either so uninformed or incompetent that they tried to fix it with a lawsuit.
- pavel_lishin 15y ago
  Or maybe one incompetent overrode them all. Manager: Evans, get in here! Why are our news articles all over The Google? Evans: Um, sir, they index our - Manager: I don't want to hear your technical mumbo jumbo! Get legal on this to get it fixed YESTERDAY! Evans: There's no need, it's literally a 5-minute fix, I could just add an entry to robots.txt while the coffee is brewing... Manager: What did I just say about mumbo jumbo? Get out of here, and send in Stevens from Legal on your way.
  - Toady 15y ago
    Again, the lawsuit was over Google News' display of lead paragraphs and other excerpts from articles. It didn't pertain to the general search engine.
    - pavel_lishin 15y ago
      The lawsuit may not have, but the ruling certainly and explicitly does.

lenni 15y ago

Maybe I just don't get it, but why do they want to be included in web search but not in Google News? Their actions seem bizarre. Both are noindex-able separately, correct?

dexen 15y ago
tl;dr: Yes they are indexed separately, and yes it's up to site owner to decide which content may be indexed for Google News -- via proper META tags and robots.txt. There is an important difference in UI between Google Web Search and Google News: the Web Search presents a very short snippet of text and link, while Google News presents a bit more of the news content. Effectively, Web Search directs viewers straight to the original website, while News lets them just browse Google News, sending less traffic to the original website. [1] Which means Google News may provide less ad revenue to original website -- probably the crux of the problem here. To handle that, Google provides two different bots: one with user agent `Googlebot', the other with `Googlebot-News'. If a website wants its content only in Web Search, and not in News, it's supposed to return proper /robots.txt -- and that does the trick. Way quicker and cheaper than any lawsuit! http://googlewebmastercentral.blogspot.com/2009/12/new-user-agent-for-news.html http://googlewebmastercentral.blogspot.com/2009/12/new-user-... ---- [1] but hey, less is more! whoever makes it to the original site via Google News has clearly show elevated interest in the news piece, and thus should be worth more to ad providers.
ChuckMcM 15y ago
Short answer, one looks like a newspaper. The longer answer is that the newspapers are struggling financially, Google is not struggling, when snippets of content appear on the Google news page the newspapers believe they are 'improving' the Google brand without being compensated. They sued to have either Google pay them, or remove them from the News site. What they fail to realize (and Google no doubt knows) is that if you aren't listed in Google you're not going to get nearly as much traffic (hence 'profit'). So Google apparently took a literal interpretation of the order and struck them from the books as it were, complete removal from the index. So the newspapers have to make the following evaluation, 'what is it worth to us to be in Google's search results?' And is that value more or less than what we think they owe us for having our results on their news page? Google clearly understands that its value for them to be in the search results and I believe they would say off the record that the value of being in the results is worth more than the ad revenue they get from the occasional story that makes it to the front page of Google news. Sort of extortion, sort of natural consequences. Humorous though.

bugsy 15y ago

OK, it says they filed a lawsuit against Google because they didn't want Google to LINK to their articles without paying a fee for linking. They won the lawsuit in court and so Google removed links to their articles from their search index, as they requested, and now they are mad about it. Why is the story "Google blocking" them. Google's only doing what a court ordered after the papers sued asking for that.

pavel_lishin 15y ago
As I understand it, they wanted Google News to stop linking to their articles; not the regular search results.
- dexen 15y ago
  From court order, via http://www.chillingeffects.org/notice.cgi?sID=2160 http://www.chillingeffects.org/notice.cgi?sID=2160 > Order the defendant to withdraw the articles (...) from all their sites (Google News and "cache" Google or any other name (...) The other articles linked from http://slashdot.org/story/06/09/18/115203/Google-News-Removes-Belgian-Newspaper http://slashdot.org/story/06/09/18/115203/Google-News-Remove... are down now, but I believe the original lawsuit was in that very spirit: remove their articles from all Google's websites. Not just from Google News. Logical argumentation aside, Google's in no position to second-guess a court order ;-) Why did Associated Press put such a negative spin to the story? We can only guess...
Toady 15y ago
They sued because Google News was using lead paragraphs from their articles. Google is most definitely block them, because the court's decision didn't pertain to the general search engine. In effect, Google is punishing these content-owners for challenging Google's use of their content in its service.

Luc 15y ago

This is not a case of technical ignorance, though it's hilarious to pretend it is. The French-language newspaper association wants to receive money from Google (for the ads displayed next to their content) and figured it would be easier to pressure them through the courts than through other means.

ToadyCensored 15y ago
The lawsuit was over the use of copyrighted excerpts from articles in the Google News aggregator and cache, such as lead paragraphs. Google is trying to pretend that the court ruled all hyperlinks to be illegal. Google was defiant before over this ruling when it ignored an order to publish the results of the verdict on its website, so it's not far-fetched to believe Google is once again throwing its weight around and reminding people of the power it wields over the livelihoods of content creators.

franze 15y ago

User-agent: Googlebot Disallow: User-agent: Googlebot-News Disallow: /

Bootvis 15y ago

The papers are back according to this news article: http://babelfish.yahoo.com/translate_url?doit=done&tt=url&intl=1&fr=bf-home&trurl=http%3A%2F%2Fwww.nu.nl%2Finternet%2F2568534%2Fbelgische-kranten-weer-google.html&lp=nl_en&btnTrUrl=Translate http://babelfish.yahoo.com/translate_url?doit=done&tt=ur... They also hint that the complaints have been lifted by the papers.

feb 15y ago
Yes, they are back in the index: http://www.pcmag.com/article2/0,2817,2388635,00.asp http://www.pcmag.com/article2/0,2817,2388635,00.asp And in a simple test, it looks to work too: http://www.google.com/search?q=copiepress+lalibre+critique http://www.google.com/search?q=copiepress+lalibre+critique The second link (lesoir.be) is one of the newspapers.