Related: https://wiki.archiveteam.org/index.php/Robots.txt https://wiki.archiveteam.org/index.php/Robots.txt (Also, consider that when you forbid such function

Related: https://wiki.archiveteam.org/index.php/Robots.txt https://wiki.archiveteam.org/index.php/Robots.txt (Also, consider that when you forbid such functionality, the only thing that happens is that its development becomes private. It's like DRM: it only hurts legitimate customers.)