3 ms·It needs to be incorporated into the robots.txt standard.by twostorytower 1y agoIt needs to be incorporated into the robots.txt standard.