4 ms·
It should start by looking at robot.txt.
by firefoxd 8mo ago
It should start by looking at robot.txt.
- malvads 8mo agoHi, thanks for your comments (it’s on the plan), since Mojo is early-stage software, there is still things that need to be integrated, however mojo is not a mass-crawler, (you have to specify directly what to crawl), so even if I add robots.txt (wich is in the plan) Evil users can still just bypass this (I expect mojo to be used by technical (non-evil) folks). But thanks for your suggestion :)