4 ms·

Web crawling by search engines shouldn't be far from web scraping in terms of data collection. I am wondering what is the legal boundary of web crawling for sea

• bytemode 7y ago
  You willingly submit links to a service to crawl your site, there's nothing like "consent" for scraping...
  • nostrademons 7y ago
    You don't, actually, most sites are discovered organically through links on other sites. Submitting links hasn't been common since the days of Yahoo and DMOZ. You're right that "consent" is the important legal issue, but it's usually implied based on what your site requires re: authentication/authorization, robots.txt, and the controls Google has provided to let you tell them not to index a site.
  • quickthrower2 7y ago
    Nope - it just requires someone to link to you. Since there are informational sites that list new domains, that might happen automatically.
• avip 7y ago
  respecting robots.txt and using publicly declared ua come to mind.