26 ms·

Is Google doing anything to solve the content duplication problem? It seems like a solvable problem. Why don't they let webmasters implement some kind of time

by staunch 16y ago

Is Google doing anything to solve the content duplication problem? It seems like a solvable problem. Why don't they let webmasters implement some kind of time-based cryptographic signature? It seems so lame that his problem has gone on so long, especially when there must be some kind of technical solution. For real businesses spending a few days implementing some authentication protocol would not be particularly burdensome.

theseanstewart 16y ago
I'm guessing a lot of it has to do with determining where the content originated. Google will crawl sites at different times of the day. If site B stole content from site A but Google came to site B first, how are they supposed to know that the content originated on site A? I'm sure there's certain things they look at like PR, trust signals, etc. to determine if the content on site B could have been copied, but it can't possibly be perfect. The time based signature sounds good in theory but implementing it across billions of pages would be very difficult. Not only that, but what if site A didn't create a signature but site B did when taking site A's content? I don't think there's an easy solution to this problem.
- protomyth 16y ago
  A human can figure it out. That's the real problem. You have a guy send in a support ticket / request, and then you pawn them off on someone who really cannot do anything about it. It really seems like support is just there to deflect rather than solve. Heck, look at the actual forum thread the article links to http://www.google.com/support/forum/p/AdWords/thread?tid=0bb4bb671eff7054&hl=en http://www.google.com/support/forum/p/AdWords/thread?tid=0bb... It just seems like a purely algorithmic solution doesn't really scale and some human intervention is really necessary.
  - forensic 16y ago
    Humans don't scale either. That's why google doesn't use humans. Sounds like the easy solution here is to have people submit their content to some google authentication system at the same time they submit to their blog. Problem solved.
    - robryan 16y ago
      They don't scale but Google can afford to hire enough to make the problem go away if they made it a focus. But not going to happen any time soon when they don't believe in customer service.
- melvinram 16y ago
  I don't know if it's an easy fixer but it's certainly not difficult to eliminate 90% of scrapers. My sites get scraped all the time and if you look at these scrapper sites, they usually are not scrapping just one site. To simplify the issue, let's look a small data set: Site 1: * Content ABC * Content DEF * Content GHI Site 2: * Content JKL * Content MNO * Content PQR Site 3: * Content STU * Content VWX * Content YZ0 Site 4: * Content ABC * Content DEF * Content MNO * Content PQR * Content STU Which of these is a scrapper?
  - lacker 16y ago
    Google does eliminate more than 90% of scrapers. It's just really easy to create scrapers; they can outnumber the original content by more than 1000-1. So you need many, many systems to remove scrapers.
  - jcampbell1 16y ago
    When you add site 5 in the mix that has: * Content ABC * Content DEF It makes it much harder to identify. If you are a blackhat SEO, then you keep track of the last time google indexes you (including the anonymous crawlers, which is tricky), and backdate scraped content to just ahead of the time you were last indexed. Then you can send a complaint through googles tools about the site that wrote the original content. Blackhats make content duplication a really challenging problem, and having a complaint form isn't going to solve much. The blackhats can take advantage of that as well.
  - Devilboy 16y ago
    The scraper is the one where the content appeared last.
    - lurker19 16y ago
      How do you perform that measurement using practically bounded computing and networking resources?
- inetsee 16y ago
  It seems to me that there should be some way for Google to see the original creators content before it goes live to the internet; some sort of pre-publishing protocol. Essentially tell the Googlebot "This is the content I am planning to publish; have a look so you will know who created it first."
  - deleted 16y ago
    [deleted]
  - lacker 16y ago
    (I used to work on Google search quality.) In general, the problem with such systems is that scraper-writers use them much more diligently than original-content-writers.
    - forensic 16y ago
      Has this been integrated into Blogspot? The trick would be getting the feature hooked into the major publishing tools.
- chmike 16y ago
  It doesn't need to do it between billion of pages. As soon as they detect duplicates, there is a high chance that it will be done again. So monitoring the two or more site with same content is doable but not very efficient. Ideally, the author subject to scrapping should be able to reference its page to google to prove anteriority. The scraper won't be able to do that.
anonymous246 16y ago
It would help with dup detection if Google had a "push" mechanism for websites to upload their content to Google.
- Psyonic 16y ago
  Not necessarily. That would only help if everyone used it. Otherwise it'd be another tool for the scrapers to claim legitimacy.
  - MichaelApproved 16y ago
    They might be able to claim it but at least it'll Also give the content creator a tool to deny that claim.
  - benologist 16y ago
    Not if you told them first - if popular CMSs/blog platforms notified Google when you clicked published they'd know just before the page even existed on the internet. Other content producers would rush to integrate it as well. I believe they already ping Google to inform them of new content, it would be a natural extension of that.
  - dodo53 16y ago
    So how about - content creator pushes content to Google, and list of people they think will scrape it. If sufficiently large sufficiently close copy of content shows up on scraper page they're penalized (adwords, search, banned from Blogger whatever). Edit: I guess that only works if they're fixed IPs.
    - Tyrannosaurs 16y ago
      You don't need the list of potential scrapers, it could apply to anyone who published the same content after. The issue though is if you write a blog and don't know to do this, the scrapers do know and nick your content and inform Google of "their" new article. At that point you're now seen as a scraper for an article you actually wrote because the real scraper plays the game better. And that's the real problem - the scrapers play the game better than the rest of us because they can commit larger amounts of time to it (not having to waste time writing actual content and all).
lacker 16y ago
Google has a lot of people working on improving search quality, and solving content duplication is a big part of that. It just isn't a trivial problem, because there are far more SEO black hats in the world working to outwit the Google algorithms.
wslh 16y ago
Yes, the time-base signature seems like part of the solution. But what amaze me is if a cryptographic signature is available some scamers will generate articles and sign them to gain reputation.

by staunch 16y ago

theseanstewart 16y ago

I'm guessing a lot of it has to do with determining where the content originated. Google will crawl sites at different times of the day. If site B stole content from site A but Google came to site B first, how are they supposed to know that the content originated on site A? I'm sure there's certain things they look at like PR, trust signals, etc. to determine if the content on site B could have been copied, but it can't possibly be perfect. The time based signature sounds good in theory but implementing it across billions of pages would be very difficult. Not only that, but what if site A didn't create a signature but site B did when taking site A's content? I don't think there's an easy solution to this problem.

protomyth 16y ago
A human can figure it out. That's the real problem. You have a guy send in a support ticket / request, and then you pawn them off on someone who really cannot do anything about it. It really seems like support is just there to deflect rather than solve. Heck, look at the actual forum thread the article links to http://www.google.com/support/forum/p/AdWords/thread?tid=0bb4bb671eff7054&hl=en http://www.google.com/support/forum/p/AdWords/thread?tid=0bb... It just seems like a purely algorithmic solution doesn't really scale and some human intervention is really necessary.
- forensic 16y ago
  Humans don't scale either. That's why google doesn't use humans. Sounds like the easy solution here is to have people submit their content to some google authentication system at the same time they submit to their blog. Problem solved.
  - robryan 16y ago
    They don't scale but Google can afford to hire enough to make the problem go away if they made it a focus. But not going to happen any time soon when they don't believe in customer service.
melvinram 16y ago
I don't know if it's an easy fixer but it's certainly not difficult to eliminate 90% of scrapers. My sites get scraped all the time and if you look at these scrapper sites, they usually are not scrapping just one site. To simplify the issue, let's look a small data set: Site 1: * Content ABC * Content DEF * Content GHI Site 2: * Content JKL * Content MNO * Content PQR Site 3: * Content STU * Content VWX * Content YZ0 Site 4: * Content ABC * Content DEF * Content MNO * Content PQR * Content STU Which of these is a scrapper?
- lacker 16y ago
  Google does eliminate more than 90% of scrapers. It's just really easy to create scrapers; they can outnumber the original content by more than 1000-1. So you need many, many systems to remove scrapers.
- jcampbell1 16y ago
  When you add site 5 in the mix that has: * Content ABC * Content DEF It makes it much harder to identify. If you are a blackhat SEO, then you keep track of the last time google indexes you (including the anonymous crawlers, which is tricky), and backdate scraped content to just ahead of the time you were last indexed. Then you can send a complaint through googles tools about the site that wrote the original content. Blackhats make content duplication a really challenging problem, and having a complaint form isn't going to solve much. The blackhats can take advantage of that as well.
- Devilboy 16y ago
  The scraper is the one where the content appeared last.
  - lurker19 16y ago
    How do you perform that measurement using practically bounded computing and networking resources?
inetsee 16y ago
It seems to me that there should be some way for Google to see the original creators content before it goes live to the internet; some sort of pre-publishing protocol. Essentially tell the Googlebot "This is the content I am planning to publish; have a look so you will know who created it first."
- deleted 16y ago
  [deleted]
- lacker 16y ago
  (I used to work on Google search quality.) In general, the problem with such systems is that scraper-writers use them much more diligently than original-content-writers.
  - forensic 16y ago
    Has this been integrated into Blogspot? The trick would be getting the feature hooked into the major publishing tools.
chmike 16y ago
It doesn't need to do it between billion of pages. As soon as they detect duplicates, there is a high chance that it will be done again. So monitoring the two or more site with same content is doable but not very efficient. Ideally, the author subject to scrapping should be able to reference its page to google to prove anteriority. The scraper won't be able to do that.

anonymous246 16y ago

It would help with dup detection if Google had a "push" mechanism for websites to upload their content to Google.

Psyonic 16y ago
Not necessarily. That would only help if everyone used it. Otherwise it'd be another tool for the scrapers to claim legitimacy.
- MichaelApproved 16y ago
  They might be able to claim it but at least it'll Also give the content creator a tool to deny that claim.
- benologist 16y ago
  Not if you told them first - if popular CMSs/blog platforms notified Google when you clicked published they'd know just before the page even existed on the internet. Other content producers would rush to integrate it as well. I believe they already ping Google to inform them of new content, it would be a natural extension of that.
- dodo53 16y ago
  So how about - content creator pushes content to Google, and list of people they think will scrape it. If sufficiently large sufficiently close copy of content shows up on scraper page they're penalized (adwords, search, banned from Blogger whatever). Edit: I guess that only works if they're fixed IPs.
  - Tyrannosaurs 16y ago
    You don't need the list of potential scrapers, it could apply to anyone who published the same content after. The issue though is if you write a blog and don't know to do this, the scrapers do know and nick your content and inform Google of "their" new article. At that point you're now seen as a scraper for an article you actually wrote because the real scraper plays the game better. And that's the real problem - the scrapers play the game better than the rest of us because they can commit larger amounts of time to it (not having to waste time writing actual content and all).

lacker 16y ago

Google has a lot of people working on improving search quality, and solving content duplication is a big part of that. It just isn't a trivial problem, because there are far more SEO black hats in the world working to outwit the Google algorithms.

wslh 16y ago

Yes, the time-base signature seems like part of the solution. But what amaze me is if a cryptographic signature is available some scamers will generate articles and sign them to gain reputation.