In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a w

In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a website for updates periodically. Technically I suppose they all should probably respect robots.txt, but in my experience something you think of as a scraper rarely does.