2 ms·
No. The text file that runs the Internet is the zone file. Robots.txt is a filtering file for the WWW.
by 1970-01-01 3y ago
No. The text file that runs the Internet is the zone file. Robots.txt is a filtering file for the WWW.
- simonblack 3y agoAnd many bots don't take any notice of robots.txt anyway.