3 ms·
Be careful about making the data you've scraped visible to Google's search engine scrapers. That's often how site owners get riled up. They search for some un
by tyingq 7y ago
Be careful about making the data you've scraped visible to Google's search engine scrapers.
That's often how site owners get riled up. They search for some unique phrase on Google, and your site shows up in the search results.
- MarcellusDrum 7y agoThis isn't really an "ethical" practice, more like how to hide that you are scraping data practice. If you have to hide the fact that you are scraping their data, maybe you shouldn't be doing it in the first place.
- tyingq 7y agoDepends. Maybe, for example, you're doing some competitive price analysis and never plan on exposing scraped things like product descriptions...you only plan to use those internally to confirm you're comparing like products. But you expose it accidentally. Avoid that.
- lazyjones 7y agoIt's incredibly ironic that one has to avoid doing what Google does in order to be kept in their index.