4 ms·
biggest scraper of the world? google. do they obey robots.txt? not a chance, they really don't care. So do what google does, which is basically they run it like
by unnouinceput 7y ago
biggest scraper of the world? google.
do they obey robots.txt? not a chance, they really don't care.
So do what google does, which is basically they run it like they own the world and guess what? it's actually legal
- pxtail 7y agoFunny thing is that it is legal for them and same rules do not apply for tiny plankton. So when it comes to scraping there is no choice but move in the gray area, hope that you don't get caught (or that they will notice too late and your product will be noticeable and large enough that it will be allowed to join big fish club)