4 ms·I had this same thought, more along the lines of expanding the usage of robots.txtby spacecadet 2y agoI had this same thought, more along the lines of expanding the usage of robots.txt