3 ms·robots.txt has a maximum relevant size of 500 kib.by ccgreg 2y agorobots.txt has a maximum relevant size of 500 kib.