4 ms·
How about a robot.txt generator. Maybe it shows the user all the directories of the website (reads the sitemap) and then allows the user to disallow specific do
by alphadown 15y ago
How about a robot.txt generator. Maybe it shows the user all the directories of the website (reads the sitemap) and then allows the user to disallow specific documents and directories in a point and click fashion. The generator then writes it in the proper syntax.
Yea? Nay?