3 ms·
I think robots.txt needs to work the other way. You put in the robots.txt ONLY if you want the search engines to crawl your page. The current way is akin to sa
by hacsky 4y ago
I think robots.txt needs to work the other way. You put in the robots.txt ONLY if you want the search engines to crawl your page.
The current way is akin to saying ... "You do not have a lock on your door. So we are welcome to come in ..."
- yakubin 4y agoThe web is a public space. There are no doors. Robots.txt is a way of saying "these are not the pages you are looking for".