3 ms·
You're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US). Google scraps
by jumperjake 11y ago
You're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US).
Google scraps webpages as a core competency.
- chinathrow 11y agoGoogle also adheres to the robots.txt standard. Most of the scrapers I block don't.
- jumperjake 11y agoNot correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.
- chinathrow 11y agoThey do not index the content but might add the URL, correct. You can have a meta noindex present and they won't index even the URL.