4 ms·

What's the difference?

• tsally 17y ago
  In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a website for updates periodically. Technically I suppose they all should probably respect robots.txt, but in my experience something you think of as a scraper rarely does.
• Raphael 17y ago
  Well, what if I just saved every HTML page I visited. I'm clearly not a bot.