4 ms·

I think this is pretty fair on Google's part. How could you possibly figure out who owned content? What if I published a book, it was copy-pasted in blogs, an

I think this is pretty fair on Google's part. How could you possibly figure out who owned content? What if I published a book, it was copy-pasted in blogs, and then later I put it somewhere crawlable by Google? You certainly can't just say "first time we saw it, that's the proper owner". It would either require a massive amount of manual QA to get right (and even then, there are going to be interminable copyright battles), or have a super high error rate. I think Google's best value is letting proper content owners easily find violators via normal searches, and let them deal with them via takedown notices or the court system -- which is where it should be done, not in a pseudo-court run by a Google who does not want what responsibility.

zaroth 11y ago
Even if you don't care about trying to identify the original source of some piece of content, it seems like the content farm site which is plagiarizing is more likely to be a lower quality site than the original content producer. The behavior does seem weird in any case, like there is a certain slot for a given piece of content, and Google is swapping different domains in and out to fill that slot. It seems like Google is actually trying to identify the original content, failing, and then actually inadvertently penalizing the original producer.
- scholia 11y ago
  Well, Google has already indexed a new article x. When article y appears, and Google sees that y is an almost verbatim repeat of x, it shouldn't be that hard to figure out that article x is the original, should it? Especially if they both have time/date stamps....
alwaysdoit 11y ago
Also all of this is assuming that any duplicated content is inherently stolen, when it could in fact be public domain, fair use, legitimately licensed, distributed under Creative Commons, etc.
- blfr 11y ago
  True, but in these cases I would still want original/canonical/fastest/best source first while the others are probably only valuable as backups.
elorant 11y ago
While most of what you say is true, they could at the very least reject AdSense applicants based on how often they copy-paste content from established publishers. I’m sure they have the means to figure something like that out.
PaulHoule 11y ago
People have been complaining about this problem for a LONG time. Duplicate removal is essential for making a web search engine that works. For instance, together with a CS research group, I built a search engine for a major university library that had more than 80 web sites. We found huge amounts of duplicate content produced by various mechanisms (for instance, multiple people posted the same stuff to the web.) If your ranking is content-based, all of the duplicate documents are going to rank the same and form a "plug" that excludes other documents. It has long (post 2006) been a common story that "I wrote a blog post but somebody else ranks for it." For instance, I made a blog post that got a huge amount of traffic in the day, but right now you search for it and you find a presentation from some fresher at Oracle that is based on those ideas. There are many factors that make this hard to control and these include: (1) for one "real" origin there are probably ten or a hundred fakes, so if you are picking at random you strike out -- you have to not only outrank one fake you have to outrank all the fakes, (2) freshness... copies are fresher than the original, also they can be updated years later, (3) also the bad guys think a lot more seriously about indexation, Page Rank, and other variables they control than do most content creators.
ChuckMcM 11y ago
So back when Blekko was a consumer search engine we could 100% figure out who owned content on sites we crawled often. And even when we didn't we could often guess correctly more often than not based on the domain registration dates. (not to mention registry owners). That is because few people who rip off content rip off just one web site, they will rip off dozens of web sites and they will all share the same AdSense ids and the same domain registrar. This is easy stuff to spot when you crawl the web regularly. I suspect that Google simply doesn't care. They get Ad revenue regardless and in their laissez-faire editorial position it doesn't matter. What are you going to do, use another search engine?
- sounds 11y ago
  Or, more likely, they can't get involved for legal reasons. If they took steps to block the easy stuff, an arms race would ensue, and the content providers would never be satisfied with the performance being provided for free by Google. The content providers would always demand stricter enforcement, and could threaten to sue for copyright infringement regardless of merit.
  - ChuckMcM 11y ago
    I think if that were the case they would not have pushed out the "Panda" updates which penalized content farms so heavily. If their past behavior (with content farms and other "low value" content sites) is a guide they will not do anything until enough people complain about it. In the mean time it isn't even Google's content so its not a hosting issue, they are just the "neutral" third party providing their 10 blue links (oh and supplying the advertising engine those sites are using)
- loceng 11y ago
  Do you know if any search engine is actively filtering for this?
  - ChuckMcM 11y ago
    Sadly no, there are only a few actual derived general search indexes (english language search) in the world, Microsoft's, Google's, Baidu's, and Yandex's. They are expensive to build and maintain and the only way to monetize them requires driving search traffic your way. Google is paying $4B/year to third parties to send search traffic their way. My guess, having been at both Google and Blekko, is that "whitelisted" search will be the next wave in the industry. For those old enough to remember Yahoo!'s original "directory" model, once Yahoo!'s contract with Microsoft is up one could hope they rebuild their search team and technology into something with a strong editorial bias for "quality" content.
est 11y ago
one man's stolen content is another man's mirror. There are countless times where original content is region-blocked or behind a paywall, or expired, but accessible via "stolen" links.

zaroth 11y ago

Even if you don't care about trying to identify the original source of some piece of content, it seems like the content farm site which is plagiarizing is more likely to be a lower quality site than the original content producer. The behavior does seem weird in any case, like there is a certain slot for a given piece of content, and Google is swapping different domains in and out to fill that slot. It seems like Google is actually trying to identify the original content, failing, and then actually inadvertently penalizing the original producer.

scholia 11y ago
Well, Google has already indexed a new article x. When article y appears, and Google sees that y is an almost verbatim repeat of x, it shouldn't be that hard to figure out that article x is the original, should it? Especially if they both have time/date stamps....

alwaysdoit 11y ago

Also all of this is assuming that any duplicated content is inherently stolen, when it could in fact be public domain, fair use, legitimately licensed, distributed under Creative Commons, etc.

blfr 11y ago
True, but in these cases I would still want original/canonical/fastest/best source first while the others are probably only valuable as backups.

elorant 11y ago

While most of what you say is true, they could at the very least reject AdSense applicants based on how often they copy-paste content from established publishers. I’m sure they have the means to figure something like that out.

PaulHoule 11y ago

People have been complaining about this problem for a LONG time. Duplicate removal is essential for making a web search engine that works. For instance, together with a CS research group, I built a search engine for a major university library that had more than 80 web sites. We found huge amounts of duplicate content produced by various mechanisms (for instance, multiple people posted the same stuff to the web.) If your ranking is content-based, all of the duplicate documents are going to rank the same and form a "plug" that excludes other documents. It has long (post 2006) been a common story that "I wrote a blog post but somebody else ranks for it." For instance, I made a blog post that got a huge amount of traffic in the day, but right now you search for it and you find a presentation from some fresher at Oracle that is based on those ideas. There are many factors that make this hard to control and these include: (1) for one "real" origin there are probably ten or a hundred fakes, so if you are picking at random you strike out -- you have to not only outrank one fake you have to outrank all the fakes, (2) freshness... copies are fresher than the original, also they can be updated years later, (3) also the bad guys think a lot more seriously about indexation, Page Rank, and other variables they control than do most content creators.

ChuckMcM 11y ago

So back when Blekko was a consumer search engine we could 100% figure out who owned content on sites we crawled often. And even when we didn't we could often guess correctly more often than not based on the domain registration dates. (not to mention registry owners). That is because few people who rip off content rip off just one web site, they will rip off dozens of web sites and they will all share the same AdSense ids and the same domain registrar. This is easy stuff to spot when you crawl the web regularly. I suspect that Google simply doesn't care. They get Ad revenue regardless and in their laissez-faire editorial position it doesn't matter. What are you going to do, use another search engine?

sounds 11y ago
Or, more likely, they can't get involved for legal reasons. If they took steps to block the easy stuff, an arms race would ensue, and the content providers would never be satisfied with the performance being provided for free by Google. The content providers would always demand stricter enforcement, and could threaten to sue for copyright infringement regardless of merit.
- ChuckMcM 11y ago
  I think if that were the case they would not have pushed out the "Panda" updates which penalized content farms so heavily. If their past behavior (with content farms and other "low value" content sites) is a guide they will not do anything until enough people complain about it. In the mean time it isn't even Google's content so its not a hosting issue, they are just the "neutral" third party providing their 10 blue links (oh and supplying the advertising engine those sites are using)
loceng 11y ago
Do you know if any search engine is actively filtering for this?
- ChuckMcM 11y ago
  Sadly no, there are only a few actual derived general search indexes (english language search) in the world, Microsoft's, Google's, Baidu's, and Yandex's. They are expensive to build and maintain and the only way to monetize them requires driving search traffic your way. Google is paying $4B/year to third parties to send search traffic their way. My guess, having been at both Google and Blekko, is that "whitelisted" search will be the next wave in the industry. For those old enough to remember Yahoo!'s original "directory" model, once Yahoo!'s contract with Microsoft is up one could hope they rebuild their search team and technology into something with a strong editorial bias for "quality" content.

est 11y ago

one man's stolen content is another man's mirror. There are countless times where original content is region-blocked or behind a paywall, or expired, but accessible via "stolen" links.