2 ms·It's not much smarter than just adding user agents to robots.txt manually.by input_sh 2y agoIt's not much smarter than just adding user agents to robots.txt manually.