2 ms·You can create a robots.txt that blocks everything but known bots (by name, like googlebot).by capableweb 4y agoYou can create a robots.txt that blocks everything but known bots (by name, like googlebot).