3 ms·
It's not like robots.txt is a great deterrent for crawlers. Only reasonable barrier would be a paywall hiding the content from the web.
by fintechie 4y ago
It's not like robots.txt is a great deterrent for crawlers. Only reasonable barrier would be a paywall hiding the content from the web.