2 ms·
You don't even need to mention robots.txt, there's plenty of people that have been sued for crawling and had to stop it and pay damages, just lookup "crawling l
by vasco 2y ago
You don't even need to mention robots.txt, there's plenty of people that have been sued for crawling and had to stop it and pay damages, just lookup "crawling lawsuits".