5 ms·
Google respects robots.txt, so it's arguably not the same as scraping a website without their consent.
by skybrian 7y ago
Google respects robots.txt, so it's arguably not the same as scraping a website without their consent.
- aledalgrande 7y agoMost sites don't have their main data/functionality in the Disallow section though.
- dodobirdlord 7y agoSure, but they could if they wanted to and that's their own business.