3 ms·Sane implementations do not read more than 500KiB of robots.txt contents.by comonoid 5y agoSane implementations do not read more than 500KiB of robots.txt contents.