6 ms·

So is SearchYC done, or do they use a scraper not a crawler?

• grinich 17y ago
  What's the difference?
  • tsally 17y ago
    In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a website for updates periodically. Technically I suppose they all should probably respect robots.txt, but in my experience something you think of as a scraper rarely does.
  • Raphael 17y ago
    Well, what if I just saved every HTML page I visited. I'm clearly not a bot.